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ABSTRACT 

Largely because of high student loan default rates, 
much of the criticism of federal aid programs has focused on policies 
and practices affecting the participation of proprietary schools in 
federal student aid. This report reviews alternative indicators of 
school performance that are currently used or could be used by the 
federal student aid programs. These indicators are considered 
appropriate for assessing proprietary schools, but they are likely to 
be applicable to other types of postsecondary institutions. The use 
of student loan default rates to set minimum standards for program 
eligibility is limited and inadequate tor assessing institutional 
quality. A more comprehensive indicator of default rates, 
particularly a default rate based on the proportion of dollars in 
default and the percentage of students receiving federal loans, would 
give a more complete picture. New information on factors influencing 
institutions likely to misuse the student aid program or to fail are 
needed to develop valid profiles to identify high-risk institutions. 
More comprehensive assessments should consider program completion 
rates, attainment of occupational competency by students # and labor 
market performance. Student background may become a factor in the 
assessment process. Specific guidelines are given for short-term 
measures to improve institutional evaluation while a more 
comprehensive assessment system is being developed. Six tables 
summarize information on indicators. Two appendices summarize 
meetings with representatives of proprietary schools and occupational 
competency testing in proprietary schools. (SLD) 
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EXECUTIVE SUMMARY 



BEYOND DEFAULTS: 
INDICATORS FOR ASSESSING PROPRIETARY SCHOOL QUALITY 



Federal student aid programs have been severely criticized because they do not have sufficient 
accountability; are often poorly managed and inefficient; and in some cases, fail to serve the 
educational needs of students. The potential for huge profits tempts some schools to engage in 
significant financial abuse. Largely because of high student loan default rates, much of the 
criticism has focused on policies and practices affecting the participation of proprietary schools 
in federal student aid. To improve these programs, the U.S. Department of Education (ED), 
Congress, and others are reexamining the basic assumptions and principles that have guided 
federal student aid policy. Current problems in federal student aid, particularly those related to 
proprietary schools, have created an opportunity to consider bold new directions. 

This report reviews alternative indicators of school performance that are currently used, or 
could be used by the federal student aid programs. Although the report focuses on 
performance indicators appropriate for assessing proprietary school performance, the 
approaches discussed are likely applicable to other types of postsecondary institutions as weU. 
In preparing this report, ED held a series of meetings with proprietary school directors, 
associations, accrediting organizations, and technical experts. 

The main findings of this report are as follows: 

Improving the Current Svstem 

The use of student Uian default rates to set minimum standards for program eligibility 
represents a limited and inadequate basis for assessing institutional quality. The default rate 
may be an appropriate measure if the sole objective is to reduce defaults, but it is inadequate 
as a measure of institutional quality. A more comprehensive indicator of defaults is preferable. 



o The elimination of institutions with high default rates from student aid, as required 
through recent legislation, constitutes a rudimentary system of performance indicators. 
Default rates have the advantage of being readUy implemented with data that ED 
already collects, and, because the d(;fault rate is a single measure, it is easily 
understood. This approach has two main problems, however: 

- First, the borrower default rate is a very limited dimension of school 

performance that fails to separate poor schooling from other factors that may 
also be associated with high defaults. 

Second, the current system uses only one measure of defaults-default rates 
based on the percentage of borrowers in default. Additional default 
indicators-in particular, a default rate based on the proportion of dollars in 
default and the percentage of students receiving federal loans-would give a 
more complete picture of the institution's experience. These indicators could 
be implemented quickly since they rely upon existing data. 



In addition to defaults, the indicators that ED routinely uses to establish institutional 
eligibility for student aid-state licensing, accreditation, and a limited number of financial and 
administrative criteria-have not worked well at ensuring program quality and protecting 
students and taxpayers from abuse. Even so, ED has not fully used the indicator information it 
already collects. 

o Reviews conducted by accrediting agencies and some state licensing agencies focus 
primarily upon indicators of school inputs-facilities and equipment, faculty training, 
curriculum, instruction methods, support services, procedures for assessing student 
progress, and space and facilities. Such indicators rely predominantly on subjective 
evidence and may be of varying quality. Previous studies of educational inputs, in 
general, have found a weak relationship between various school inputs, when they are 
measured quantitatively, and independent measures of school effectiveness. 

- Some accreditation agencies have collected data on student outcomes, but 
have not developed performance standards or tied results directly to 
accreditation decisions. 

- Improvements are not likely to come from new measures of school inputs, or 
from efforts to quantify their measurement. Although there are important 
issues regarding the function of accreditation and state licensing in student aid, 
they are beyond the scope of this report. 

o Indicators of financial and administrative capacity currently employed are important to 
protect students from insolvent institutions, and to protect taxpayers from abuse of 
student aid funds-but their effectiveness, to date, is quite limited. Although more must 
be done with existing data to detect financial abuse and enforce existing rules of 
program eligibility, it is also necessary to obtain data that are more current and 
comprehensive, and to use these data more efficiently. Changes needed include: 

- More timely data. Because federal financial losses can mount rapidly once schools 
enter the student aid program, it is unwise to rely on financial information that is, 
in many cases, 18 months or more out of date. More timely data on Guaranteed 
Student Loan (GSL) volume, defaults, and percentage of students on financial aid 
are needed. 

- Development of statistically valid, high-risk profiles to identify institutions that 
are potentially insolvent or likely to misuse the student aid program. These 
profiles, based on a random sample of schools, could be used to improve the 
targeting of program reviews and investigations. Use of high-risk profiles to detect 
potential financial abuse would be patterned after the approach used in other 
federal programs that involve financial transactions with many vendors (e.g., Food 
Stamps, Medicare). 

Collection of new information on factors such as the adequacy of financial 
arrangements if the school fails, percentage of "ability to benefit" students 
enrolled, percentage of school revenues derived from federal student aid, 
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percentage of school revenues spent on recruitment and other non- 
instructional functions. 



Although limited implementation of these recommendations ';an begin with existing 
data, new data requirements and statistically valid indicators could take two or three 
years to develop and fully put in place. 

A romprchensive Approach 

In addition to improvements in the current set of performance indicators, the student aid 
system should move toward a more comprehensive assessment of school performance. This 
approach would use multiple indicators-defaults, Hnancial abuse, ai d student outcomes-to 
obtain a more complete picture of institutional quality and performance. 

o A comprehensive assessment of performance should have several objectives: 

- to set minimum standards of eligibUity in order to effectively screen out schools that 
are abusing the programs or providing ineffective education and training to students; 

to create incentivcL' to bnprove uistitutional performance and effectiveness for those 
schools exceeding minimum eligibility criteria; 

- to enhance consumer choice by providing prospective students with useful information 
about the performance of other students attending an institution; and 

- to generate data that policymakers can use to assess performance of the entire system 
and to gauge improvement. 

To date, most of the discussion concerning student aid has focused on the first objective-setting 
minimum eligibility standards. A comprehensive performance based system must go beyond 
setting minimum standards to also incorporate these other objectives-creating incentives for 
program improvement, consumer choice, and system assessment. 

o Such a system should uiclude, for proprietary school students, outcomes related to 
program completion, attainment of appropriate levels of occupational competency, and 
labor market performance. 

- The student aid programs currently reward schools based on enrollment, and 
consequently, encourage schools to admit students regardless of whether they are likely 
to benetit from the training that is provided. Program accountability that is heavily 
based on student outcomes is consistent with policy directions in the Job Training 
Partnership Act (JTPA), Vocational Education, and other federal job training 
programs. 

- Program completion, student learning outcomes (e.g., passing state 
licensure/certification exams, other tests of occupational competency), and labor 
market outcomes (e.g., training related placements, job retention, wages, employer 
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satisfaction) are all important indicators by which proprietary school performance may 
be measured. A balanced ^stem should include measures in each of these areas. 

1. Completion rates are a key measure of student attainment. 
Completion rates are important, for studies have shown that students 
who complete are much more likely to derive economic benefits from 
their training than those who do not. 

2. Learning outcomes assess the extent to which students have acquired 
adequate knowledge and skills to enter and progress in their chosen 
fields. This outcome is important to ensure that schools have actually 
trained students, and not simply provided very expensive job 
placement assistance. 

3. Labor market outcomes assess the eirtent to which training programs 
have achieved their principal objective~to help students obtain jobs. 
Employment 13 weeks after program completion and weekly earnings 
are two of the basic performance indicators used in JTPA. 

Assessment of schools on the basis of student outcomes is a long-range undertaking 
since none of the information required to measure student outcomes is currently 
available to ED. It is feasible for ED to establish data systems to obtain information 
on student outcomes, for several states already have elements of such a system in 
place. 

Special indicators could be developed to gauge how well schools serve populations of 
particular interest. One set of indicators should compare completion rates, placement 
rates, and te^t score results for "ability-to-benefit" students and high school graduates 
to indicate whether "high risk" students are admitted because they are likely to benefit 
from their training, or just because of the financial aid they provide. Another set 
could compare student outcomes for recipients of federal financial aid and unaided 
students. 

ED may wish to develop an atyustment methodology that considers student background 
characteristics when assessing institutional performance regarding defaults and student 
outcomes. To ensure that lower performance expectations for certain groups of students 
are not created, performance standards should be established that require schools to 
substantially exceed-not just meet-adjusted performance levels for high risk students. 
This approach would result in realistically attainable standards while providing 
incentives for program improvement. 

Experience suggests that performance indicators may create unfair compavisons 
among schools serving different types of students, or may encourage creaming (the 
enrollment of students who are least likely to default or easiest to place in jobs). 



Setting standards requires data on patterns of prior institutional performance; such 
data are not currently available to ED. 
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Next Steps 



Development of a fully operational performance based accountability system will take several 
years. Some indicators of student performance may be phased in sooner than others. In the 
meantime, ED must do more with currently available data on defaults and flnancial abuse, and 
develop enhanced measures in both of these areas. 

o In the short run, ED can develop multidimensional default rate indicators. 

These rely entirely on existing data and can therefore be quickly implemented. 

These indicators could be combined in a composite default index. 

o Although ED can do much to improve its use of existing data to identify flnancial risks 
and potential abuse, fiill implementation of recommendations to improve the timeliness 
of data on schools' flnancial performance, expansion of the range of flnancial indicators, 
and development of "risk proflles" to target investigations will take several years. 

High risk profiles can be developed with existing data, but more comprehensive and 
timely data will enhance their validity and effectiveness. The establishment of 
improved data systems will take several years. 

o Indicators of student performance can be phased in as data becomes avaUable. The 

Student Right to Know Act requires schools to report completion rates beginning in 1993, 
and ED is conducting feasibility studies to determine whether job placement and pass 
rates on state licensing exams can be obtained as well. Much additional work will be 
required to define outcomes, develop measures, specify standards, establish data 
collection systems, and ensure the accuracy of information collected. To help achieve 
these long-term goals» ED should: 

Establish an external "performance indicators panel" of educators and technical 
experts to advise ED on the appropriateness of alternative indicators, measures and 
standards; the use of adjustment methodologies in assessing institutional performance; 
the feasibility of implementing data collection activities; and coordination with other 
accountability systems at the state or federal level. 

Sponsor a limited number of demonstration projects with states and 
accreditation agencies tc test the feasibility and effectiveness of different 
approaches. Implementation of performance-based policies in student aid 
will require, in most cases, that ED devise appropriate roles for states and 
for accreditation agencies. 

Create a joint initiative that includes the Department of Labor and the Department 
of Health and Human Services to address common i<«ues encountered in 
performance based policies for vocational and job traming programs. Such an 
initiative could design better ways to measure occupational competencies; establish 
procedures for using state wage records to determine whether program completers 
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are employed; and deveU^p objective critcjria for deciding when a job placement is 
••training related.^^ 

To develop a performance measure that separates school performance from student 
characteristics, an option is to adjust gross performance measures on the basis of student 
background characteristics. This approach would compare performance at schools serving 
similar populations, and hold schools accountable where performance fell below a level 
predicted on the basis of student characteristics. 

Some of the data needed to begin developing such models are available or can be 
obtained; betler data will be available in two years. Stuc^ent background 
characteristics that may be considered include family income, percentage of ••ability to 
benefit^^ students, and if available, a measure of entry level academic preparation. 
Other factors to consider are local labor market conditions and average entry wages 
in the field of training. 

Recognizing that the use of an adjustment methodology in studenr aid might be 
controversial, a performance indicators panel (described above) could be asked to 
examine the issue. The panel could begin by examining how well adjustment methods 
have worked in other programs and the extent to which promising methodologies 
developed in JTPA, in state education agencies, and in health care delivery could be 
applied to student aid. 
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Beyond Defaults*. 
Indicators for Assessing Proprietary Sch ool Quality 

Introduction 

To assess the effectiveness of proprietary schools that 
participate in the federal student aid program, the U.S. 
Department of Education now relies primarily on student loan 
default rates, along with some additional information about 
financial and administrative capability and state licensing and 
accreditation agency reviews. These arrangements have been 
widely criticized because they have not excluded abusive or 
ineffective institutions; moreover, they contain no incentives 
for schools to improve program performance. This study was 
undertaken to consider alternatives to relying primarily on 
student loan default rates as an indicator of school performance. 
Although the approaches discussed here could also apply to other 
types of postsecondary institutions, this report is limited to 
proprietary schools— where the most serious questions about 
participation in student aid and institutional performance have 
been raised. 

Proprietary Schools 

Proprietary schools are for-profit organizations that 
generally provide short term occupational training. There are 
about 6,000 proprietary schools in the United States, two-thirds 
of which are separately accredited main campuses; the remainder 



1. For a detailed discussion of proprietary institutions and students, see Richard Apling, "Proprietary Schools: A Descriptii 
1 of Institutions and Students" (Washington. DC; Congressional Ftesearch Service, 9(M28 EPW) 1990. 
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are branch campuses. About 5,000 proprietary schools participate 
in the federal student aid program. Proprietary schools 
represent more than 60 percent of all institutions currently 
certified for student aid. 



Most proprietary school training is concentrated in office 
occupations (word processing, secretarial work, bookkeeping) , 
cosmetology (hairdressing, barbering) , and technology training 
(computer science, data processing, electronics). According to 
the Congressional Research Service, 64 percent of all proprietary 
schools offer training in cosmetology, business, and marketing. 



Proprietary schools are of special concern in the context of 
federal student aid policy for several reasons 

o They depend heavily on federal student aid: 81 percent of 
their students received federal student aid in 1986, 
compared with about 35 percent for all undergraduate 
students; 

o The cost to the federal government of providing such aid 
is high: the average amount of federal aid received by 
proprietary school students in 1986 was $3,400, compared 
with $2,100 for students at similar public vocational 
institutions, $1,700 for aided students attending 
community colleges, and $2,666 for undergraduates, in 
general ; 

o Proprietary school students frequantly finance their 
training with loans: 67 percent of proprietary school 
students received Stafford Loans in 1986, compared with 
about 20 percent among all undergraduates; 



2 The information in the following four bullets is from the same source: John Tuma. Antoinette Gifford, and Susan Choy. 
"Student F: lancial Aid and Postsecondary Vocational Education." prepared for the National Assessment of Vocational Education. 
(USED, Washington, DC, MPR Associates, 1989). 
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o Proprietary schools serve a substantially less affluent 
population (58 percent of proprietary school students 
had incomes under $23,000 in 1986, compared with 36 
percent of all undergraduates) ; and 

o Students attending proprietary schools have a higher 
propensity to default (27 percent for the 1988 student 
cohort) on federally guaranteed loans, compared with 17 
percent for all students.^ 

High defaults have been regarded as one symptom of a more 
general problem with institutional quality and effectiveness; 
hence, in recent years reducing default rates has been the object 
of substantial effort by the Department of Education and 
Congress. The most recent attempt to reduce defaults curtails 
eligibility for federal student aid among schools with default 
rates exceeding 35 percent in three consecutive years. This 
requirement became effective on August 15, 1991. Most of the 
institutions affected are proprietary schools. 

Performance Based Systems 

Although performance based systems cover a wide array of 
different policy options, all share three features: (1) a central 
role for measures or indicators of how well or poorly the 
suppliers of education and training perform; (2) standards that 
establish an expected level of acceptable performance; and (3) a 
"feedback" mechanism through which information on past 
performance is used to make future performance better and 
suppliers are held accountable for their performance. 



3. ED analysis of GSL records in official "tape dump." 

3 
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The basic premise underlying performance-based policy is that 
information on the performance of institutions, suitably tied to 
rewards, sanctions, funding, and dissemination, can improve a 
federal program in two ways: through the reallocation of federal 
resources away from ineffective suppliers and toward more 
effective ones, and through the efforts of individual schools and 
students to improve. 

Performance-based policies will take several years to design 
and implement fully. Multiple goals must be sorted out and 
ranked by priority. Some indicators are easily measured with 
available data, whereas others will require the development of 
new measures and data collection systems. Although ultimately a 
system that employs multiple measures and serves several 
objectives is highly preferable, creating such a system may not 
be possible in the short run. 

This report describes the choices to be made among alternative 
goals and indicators that could constitute a performance-based 
system for federal student aid. Some strategies discussed in 
this paper are based on the experiences of the Job Training 
Partnership Act (JTPA) and other employment and training 
programs. Others are based on the efforts of states to introduce 
performance indicators into their educational systems, and the 
use of indicators in other federal programs to detect potential 
financial abuse and error. 
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In preparing this report, ED staff have met with 
representatives of the proprietary school community to discuss 
their views on the types of indicators that are most appropriate 
for these institutions. (Appendix A summarizes the discussion 
that occurred at meetings with representatives from three 
occupational fields— cosmetology, electronics, and secretarial 
work) . In addition, ED held several informal meetings with 
representatives from the various proprietary school associations 
and accreditation agencies. 

The performance indicators considered here are those deemed 
appropriate for proprietary schools that aim principally to 
provide occupational training. Although the approaches discussed 
here may be applicable to other types of institutions, (e.g., 
public institutions that offer occupational training similar to 
that found in proprietary schools) the merits of applying them 
more widely are not addressed here.* 

Finally, this report makes two basic assumptions about future 
policy. First, despite some recent proposals to curtail 
eligibility of proprietary schools for student aid, this report 
assumes that these schools will remain eligible for student aid. 



4 The limitation to proprietary schools considerably simplified our task. Although most of the performance indicators that 
mal<e sense for these schools also could apply to public sector institutions, the difficulty lies in the detail of measurement. 
Because community colleges offer academic as well as occupational programs of study, efforts to assess labor nriarket or 
learning outcomes depend on proper classification of students as engaged in "occupational training" or "academic/transfer 
programs." Similarly, students may enroll to earn a degree or certificate, but significant numbsrs do not seek degrees. Aiiy 
"completion rate" measure must take account of the differences in student objectives. These are not significant issues in the 
proprietary sector. 
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Second, although the Department of Education has only recently 
undertaken a major effort to overhaul the management structure 
that administers student aid, it ir assumed that the options 
outlined in this paper will be compatible with, and could 
constitute a component of, any plausible future management 
approach. 

Alternative Objectives of Performance Indicators 

An indicator measures a student, institution, group of 
institutions, or the entire system along dimensions considered 
significant for judging performance. Because most performance 
consists of complex behavior, indicator systems are usually made 
up of several measures that tap different dimensions of 
performance. 

Indicator systems may be set up to achieve several different 

objectives. Although these objectives are not mutually 

exclusive, specific design characteristics of an indicator system 

depend heavily on the most important objectives. Indicator 

systems may be established for several purposes: 

o To set minimum eligibility standards . In systems that 
set minimum standards, schools falling below a specified 
standard may be subject to a loss of funding or 
additional oversight requirements. The exclusion from 
student aid of schools with defaults exceeding 35 
percent is an example of how indicators may be used to 
set minimum standards. There are relatively few 
examples in public policy where performance measures are 
used to apply severe sanctions such as those planned for 
student aid; Florida requires a 70 percent job placement 
rate in its public vocational programs as a condition of 
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state funding, but it is not clear that programs have 
actually been "defunded." 

To improve institutional performance . Performance 
indicators may encourage individual institutions to 
improve their performance. Through information 
disseminated to policymakers, schools, and licensing and 
accrediting organizations, institutions may be 
encouraged to make changes in curriculum, teaching, 
student counseling, and job placement. Indicators may 
also be tied to the distribution of rewards. 
Institutions that score high on measures of perfoinnance 
or exceed a specified standard receive more funds, and 
those that score low or fall below a standard may have 
their funding reduced. Examples of the use of financial 
rewards are JTPA's 7 percent performance incentive funds 
and Tennessee's 5 percent performance funding for higher 
education. Financial incentives usually involve 
marginal shifts in funding. Rewards may also take the 
form of regulatory relief; that is, schools that 
substantially exceed a standard are subject to less 
burdensome audit, review, and monitoring requirements. 

Systems designed to improve performance focus on change 
over time, not just on the measure of performance during 
one time period. Measures that gauge change over time are 
particularly important for assessing improvement. 

To improve program monitoring . Experience suggests that 
some types of indicators are highly correlated with forms 
of institutional behavior that violate federal policy. For 
example, indicators are typically employed in federal 
programs to identify high-risk vendors or individuals most 
likely to engage in financial abuse. Such sources may then 
be targeted for in-depth program review. Indicators do not 
constitute evidence of abuse per se, but simply improve the 
targeting of investigations and, ultimately, sanctions that 
are imposed. 

To produce consumer information . Dissemination of 
information to students, parents, and employers, can have 
a powerful effect on the demand for high-quality 
training. Armed with information on completions, job 
placements, and earnings, students can shift their 
enrollment toward suppliers with superior performance. 
Because "money follows the students"— that is, revenues 
from student aid and tuition are directly linked to 
enrollment — there could be a strong incentive for 
programs to raise performance. In contrast to systems 
that measure improvement through "value added" and 
"change over time" indicators, consumer information 
systems rely more heavily on measures of gross 

7 



17 



performance during a recent time period. Thus a 
prospective student cares little about whether a 
school's job placement rate is improving or getting 
worse, but only whether the student is likely to get a 
job related to the training received. 

o System assessment . Although the primary purpose of most 
indicator systems is to assess the performance of sub- 
national components — programs, institutions, or states — 
the information may be aggregated to assess the 
performance of the entire system and to assess change 
over time. Such information can be important to federal 
program managers. Of course, aggregation to the 
national level requires comparability of data across 
occupational fields, types of institutions, and states. 



Accounting for Student Characterist ics; Adiustincf Outcomes 

Holding institutions accountable for the performance of their 
students could result in institutions serving those students with 
the greatest potential to succeed, and not serving higher risk 
students who are most in need. Adjusting expected outcomes for 
the characteristics of the students would reduce the likelihood 
of creaming; that is enrolling students who are least likely to 
default on loans or are easiest to place in jobs. On the other 
hand, adjusting for performance outcomes could institutionalize 
lower performance expectations for certain groups of students, 
and reduce the incentive for schools to recruit students wisely — 
that is, to select students who are most likely to benefit from 
the training received. 



Other federal programs have recognized the importance of 
addressing the potential for creaming—notably JTPA and 
Vocational Education. JTPA has developed a regression model that 
adjusts performance scores to corsider multiple student 
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characteristics and local labor market conditions. The recently 
reauthorized Carl Perkins Vocational and Applied Technology Act 
requires states to develop standards and measures of performance 
for secondary and postsecondary vocational education. Congress 
specified that such systems contain "incentives or adjustments 
that are designed to encourage service to targeted groups or 
special populations" (P.L. 101-392. Sec. 115). 

Two basic approaches exist to adjust for the characteristics 
of students: 

o one approach is to measure the "net impact" of a program 
on students. In this case, performance measures would be 
adjusted fo:: participants' skills, knowledge, and 
experience before entering the program and would hold 
institutions accountable for improvement. This approach 
would determine net impact by comparing expected 
performance for a student in the absence of the program to 
actual outcomes after leaving school. Statistical 
techniques to predict expected performance would provide 
net impact estimates. 

This approach requires data on program participants and 
non-participants that could be periodically collected 
for a sample of institutions. "Net impact" could be 
measured through an evaluation conducted on a sample of 
institutions, but it is not feasible to implement this 
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approach through the use of regularly collected 
institutional records. 

Another approach is to predict performance (e.g. defaults, 
job placements, etc.) for institutions based on the 
characteristics of their student body and then hold 
institutions accountable for doing better than other 
institutions serving similar types of students. This 
approach measures the "relative effectiveness" of one 
institution compared to others. 

This approach would build existing levels of performance 

into the base and therefore, could institutionalize poor 

performance. To avoid this, performance standards should 

be set at levels that require schools to substantially 

exceed the performance level predicted on the basis of 

student body characteristics. Setting standards at a 

level that requires schools to exceed substantially the 

performance level predicted on the basis of student 

characteristics will result in demanding, but realistical7ty 

attainable, expectations. This approach can be a powerful 

tool for program improvement. There are several ways in 

which this approach could be implemented: 

Comparisons among schools serving similar populations 
of students . Schools could be rank ordered on an 
index that measures socioeconomic status, and school 
performance could be compared among similarly ranked 
institutions. Several states (Kentucky, California, 
and South Carolina) employ this approach in assessing 
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reading and math achievement scores of elementary and 
secondary students.^ 

— Statistical adjustment models that predict 
performance based on student chara cteristics. JTFA's 
adjustment model is based upon the previous 
experience of participants in the program. It 
considers multiple student characteristics and local 
labor market conditions.^ In the context of student 
aid, this approach would compare a predicted level of 
school performance (e.g., defaults, placements, etc.) 
to the actual level of performance. Performance 
outcomes at a school would be predicted on the basis 
of as many relevant characteristics as available data 
will permit (e.g., family income, percentage of non- 
high school graduates/ "ability to benefit" (ATB) 
students, academic ability). The difference between 
the predicted performance and the actual level may be 
attributed to the school. Schools could be penalized 
for poor performance or rewarded for performance 
substantially above the predicted level. 

— Measures of "value added" that compare per formance 
be fore the program to perform ance after the program. 
This approach would measure change in performance 
during the time that students were enrolled; 
comparing institutions would provide a measure of 
relative institutional effectiveness. This approach 
is appropriate where it makes sense to develop pre- 
and post-measures of performance, e.g., gains in 
basic skills or occupational knowledge. It would be 
potentially misleading to rely exclusively on this 
approach, however, because: a) measures of pre- 
program employment and wages are inappropriate if 
students were not previously in the labor market 
seeking employment (e.g. recent high school 
graduates, housewives) ; and b) measurement of post- 
program improvement is only feasible for students who 
complete the program, but not for those who withdraw 
early. 



5. Joseph Wholey. "Using Evaluation to Improve Program Performance," University of Southern California, Washington Public 
Affairs Center, forthcoming. 

6 The JTPA adjustment model considers such barriars to employment as school dropout incidence, limited English 
language proficiency, handicapped incidence, offender status, reading skills below the seventh grade level, math ski ls below 
the seventh grade level, long-term AFDC receipt, lack of a significant work history, homelessness, substance abuse, teen 
pregnancy/parenthood, or JOBS program participation. 
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The data and systems needed to adjust for all relevant student 
characteristics do not currently exist, but they could be 
developed. In developing an approach to reduce creaming while 
also holding institutions accountable for improved performance, 
the second approach — the one emphasizing relative impact of 
institutions, is the one that is most feasible to implement using 
regularly collected program records. 

Alternative Indicators of Performance 

Four broad types of indicators appear relevart to proprietary 
schools: (1) default rates; (2) other indicators of financial and 
administrative abuse or instability; (3) indicators of program 
inputs and process; and (4) indicators of program effectiveness 
or outcomes. The Department of Education now collects 
information on student defaults and on a limited range of 
financial and administrative features. State licensing and 
accreditation agencies determine quality by focusing on program 
inputs and procedures. The Department does not collect any 
information with which to assess institutional performance based 
on student outcomes. 

Defaults 

Defaults reflect the frequency with which student borrowers 
under the Guaranteed Student Loan (GSL) program do not repay 
their loans on time. School default rates are based entirely on 
the proportion of borrowers in default. ED makes no adjustment 
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to account for students who repay late, for institutional 
differences in dependence on federal loans, or for student 
characteristics . 

schools that have high default rates have been associated with 
several problems of legitimate concern to policymakers: 
financial abuse, organizational instability, administrative 
impairment, and programs of poor educational quality. All these 
concerns were recently documented by the Nunn Committee 
investigation of federal student aid programs. The committee 
found "overwhelming evidence that the GSLP (Guaranteed Student 
Loan Program), particularly as it relates to proprietary schools, 
is riddled with fraud, waste, and abuse, and is plagued by 
substantial mismanagement and incompetence. Hundreds of 
thousands of students have been left with little or no training, 
no jobs, and significant debts that thev cannot possibly r epay 
(emphasis added). Moreover, the cost to the federal 
government of defaults has risen so rapidly that it now exceeds 
50 percent of total annual federal costs for the GSL program. 
The default rate indicator is attractive to program 
administrators because it can be readily calculated from existing 
data. Becausf?. no additional information is required, the 
likelihood of swift action to remove high-default schools from 
the federal student aid program is increased. 



7. U.S. Senate, "Abuses in Federal Student Aid Programs," Report of the Committee on Govemmental Affairs. Permanent 
Subcommittee on Investigations, (Washington, DC: US Government Printing Office, 1991) p. 6. 
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But use of student default rates as the primary measure of an 

institution's performance may be problematic for several reasons: 

o Default rates may be less directly and consistently 
associated with poor program quality or financial abuse 
than is frequently alleged. According the Congressional 
Research Service: 

"Concerns over proprietary school participation in Title IV 
programs are rooted in two major problems: apparent 
profiteering from Title IV by some unknown number of 
schools through fraudulent and abusive practices; and high 
GSL default rates by proprietary school students. There is 
debate over the extent to which these problems may be 
interrelated — that is, that school abusive practices are 
causing high defaults — but there is general agreement that 
they exist and should be addressed to assure the future 
viability of title IV programs."^ 

Preliminary results that compare default rates with 
independently collected employer ratings of program quality 
suggest a weak correlation. According to data collected 
for the National Assessment of Vocational Education (NAVE) , 
which contain employer ratings for 28 proprietary school 
programs in three different cities, default rates for the 
lowest-rated institutions (18.5 percent) were essentially 
the same as those for mid rank (12.8 percent) and highly 
rated institutions (18.1 percent).' 

Another study compared high default institutions with a 
different measure of program quality — approval by the 
Veterans Administration for GI educational benefits. Of 
the 84 high default institutions considered, 50 percent 
were approved by the VA, 20 percent had their approval 
withdrawn, and the remainder never participated in VA 
programs . 

o School default rates do not separate school-related factors 
from other important factors that are also associated with 
defaults. In addition to the type of institution attended, 
default rates are strongly associated with student 



8. Charlotte J. Fraas, "Proprietary Schools and Student Financial Aid Programs: Background and Policy Issues" 
(Washington, DC; 90.427 EPW, Congressional Research Service, 1990), p. 19. 

9. These findings should be regarded as simply suggestive because of several limitations in the data. (1) the sample is 
small; (2) employer ratings applied to individual programs-not to the entire institution; and (3) the sample is limited to programs 
in accounting and manufacturing, rather than including fields with the highest default rates. 

10. Valentina Tlkoff, "Screening Schools: How the Departments of Education and Veterans Affairs Approve Institutions for 
Participation in Student Assistance Programs," US ED. Planning and Evaluation Service, 1991. 
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background characteristics, and with postschool earnings 
(which, in part, reflect the field of study) . Research on 
the factors associated with loan defaults indicates that 
race, family income, and low wages (relative to other 
postsecondary students) are strong predictors of student 
defaults. " A recent review of default rate studies by 
the General Accounting Office identified the following 
factors, in addition to attending a vocational/trade 
school, associated with defaults: low income, little 
financial support, minority background, lack of a high 
school diploma, failure to complete educational program, 
enrollment for less than one year, small amount borrowed, 
and unemployed when defaulting 

The use of absolute default rates as a condition of 
eligibility for student aid could therefore penalize 
those schools that enroll students with a higher 
propensity to default: minorities, economically 
disadvantaged students, and students whose training is 
in low-wage fields. 

Another factor that could also affect a school's default 
rate is the quality of loan servicing. A high-default 
school, for example, could have loans that, but for poor 
servicing, might not have entered default. 

o The definition ED uses to measure default rates may not 
fully reflect the extent of the problem. The 
Department's definition does not consider whether 
repayment has occurred after a student enters default, 
the number of borrowers at an institution (high defaults 
based on few loans is clearly of greater concern than 
high defaults based on many borrowers) , the proportion 
of dollars borrowed that are in default, and the 
percentage of students enrolled who borrow (schools 
F'ri^re everyone borrows might be asked to incur special 
obligations compared with schools where a fraction of 
the students obtain federally guaranteed loans) . 

o Default rates are currently based on the percentage of 
loans that ever enter default, but many loans, although 



11. Mark Dynarski, "Analysis of the Effect of School Type and Post-School Earnings on Default," prepared for U.S. 
Department of Education, Planning and Evaluation Service, by Mathematica Policy Research, forthcoming. See also Laura 
Knapp and Terry Seaks, "An Analysis of the Probability of Default on Federally Guaranteed Student Loans" (Washington, DC: 
College Board, 1990). Although Dynarski found that the type of institution attended (proprietary schools and community 
colleges) also affects the propensity to default, no carefully controlled studies have established the relationship between defa 
and the specific Institution attended. Knapp and Seaks's study found no relationship between type of institution and default. 
Their study did not Include proprietary schools. 

12. General Accounting Office, 'Student Loans: Characteristics of Defaulted Borrowers in the Stafford Student Loan 
Program,' GAO/HRD-91-82BR, 1991. 
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initially in default, are subsequently repaid. One 
study estimated that approximately 30 percent of 
defaulted loans enter repayment within 20 months after 
default, and most loans that enter repayment do so 
within the first few months after default." This 
study estimated that 40 to 45 percent of defaulted loans 
would enter repayment within 36 months. Such loans are 
treated as in default, however, when ED calculates 
default rates. This practice produces a "worst case" 
estimate of defaults that may overstate the extent of 
the problem. 



supplemental Measures of Defaults . The use of multiple 
measures or indicators of performance provides a more complete 
picture than does reliance on a single indicator. Current 
practice focuses on only one dimension of the problem: the 
frecmencv of defaults. This approach does not distinguish 
between schools with few borrowers and those with many borrowers, 
and schools whose students borrow small amounts and schools where 
the loans are more substantial, schools in which most students 
have Stafford Loans and schools where loans are not a significant 
source of institutional finance. Use of several dimensions — 
frequency of defaults, proportion of dollars in default, and 
percentage of students who borrow would provide a more complete 
picture of the default rate problem at individual schools than 
any single dimension does. 



Creation of a composite default index using all of the above 
mentioned measures— frequency of default, proportions of dollars 



13. American Management Systems, "Stafford Student Loan Program Pilot Debt Collection Study." prepared for U.S. ED, 
1990. 
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in default, and percentage of borrowers— is one option by which 
multiple dimensions of the default problem can be considered. 
Combining such measures and weighting them as appropriate could 
produce a school's default index. Standards of school 
eligibility for student aid could be established in accordance 
with an institution's default index score. 

The development of a composite default index could be 
undertaken in a relatively short period of time—possibly six 
months. All the information needed is available or can be 
calculated from existing data (see table 1). 

The second major problem with the use of a gross default rate 
is that it is not sensitive to factors that have a powerful 
independent effect on defaults, such as the socioeconomic 
background of the students served. One way in which gross 
default rates may be made a more useful indicator of school 
performance is to employ a statistical model to adjust default 
rates based upon the characteristics of the students that borrow. 
Such an approach would compare the gap between actual defaults at 
a school and a level predicted on the basis of student 
characteristics , 

Data availability is one obstacle to immediate implementation 
of an adjustment model approach (see Table 1). Ideally, it would 
be helpful to have information for all proprietary schools on a 
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Table 1 
Default Rate Indicators 



Measures 



School default rate 



Data Needs 
Borrower default rate 



Availability 
of Data 

Yes 



Use 

Currently used to 
establish minimum 
eligibility require- 
ments . 



Composite default index 



Ad-iusted defaults 



Borrower default rate 
Dollar default rate 
Percent of students 
who borrow 

Borrower default rate 
Family income 
Ability to benefit 
Expected wages 
Academic Ability 



Yes 
Yes 

Yes 1/ 



Yes 
Yeia 
No 2/ 
Yes 3/ 
No 



Not used 



Not used 



1/ Information on percentage of students who borrow may be derived using enrollment data 
in IPEDS. 

2/ Information on the percentage of non-high school graduates enrolled (not the percentage 
of non-high school graduates receiving federal student aid) appears in the application 
for instit.itional eligibility and certification. This information is not routinely 
updated . 

3/ Available from Department of Labor. 
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wide range of student characteristics: family income, high 
school graduation status, dependency status during repayment, 
expected earnings for the field of study, and entry-level test 
scores. Unfortunately, the main source of program records, the 
GSL tape dump, contains only data on income and dependency status 
before borrowing. To implement this approach fully would require 
augmenting data on the GSL tape dump— a process that could take 
up to several years. 

in the interim, a more limited model of predicted defaults 
could be developed using existing data on the GSL tape dump 
(primarily family income and dependency status) merged with data 
published by the Bureau of Labor Statistics on entry level wages 
for different occupations. 

Another short-term option is to compensate for the data 
limitations of program records by estimating defaults using 
existing data sets that contain more complete information on 
student characteristics. Two possibilities are: the National 
Postsecondary Student Aid/Student Loan Recipient Survey and 
various state data bases. Values derived from these data sets 
could then be applied to individual schools using student record 
data that schools v/ould be required to provide or data that is 
available elsewhere (e.g., Integrated Postsecondary Education 
Data System — IPEDS) . Longitudinal data from the National 
Postsecondary Student Aid Survey of Beginning Postsecondary 
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students will be available in 1993; this could be merged with the 
GSL tape dump to develop an adjustment model predicting defaults. 
The data could also be used to estimate other student outcomes 
for students enrolled in short term programs. 

Indicators of Financial Abuse and P rogram Stability 
If nothing else, student aid is a large financial network in 
which substantial amounts of money move between the government, 
students, service providers, banks, and other intermediaries. 
Such systems contain ample opportunity for abuse by anyone out to 
make profits at the expense of students and the taxpayer. High 
default rates may be one indicator of financial abuse. Financial 
fraud and errors may take many other forms as well — for example, 
unethical recruiting practices, failure to implement tuition 
refund practices, failure to notify lenders of changes in 
registration status, and complicity in submission of inaccurate 
aid applications, etc. 

Defaults and financial abuse may or may not be related to one 
another. Eliminating high default schools from Title IV may not 
adequately address concerns about financial abuse. Constant 
monitoring to detect and deter fraudulent and abusive behavior is 
mandatory . 

A closely related concern is program stability. Marginal 
institutions that go out of business leave their students with 



serious financial and educational problems. Students are left to 
repay loans for education and training that was never provided. 
Students who default on these loans are barred from future 
borrowing to finance subsequent education, and of course 
taxpayers must assume any defaulted debt. There are no 
systematic data on the extent to which students are left to repay 
loans incurred at institutions that "folded"; this problem should 
be investigated further. 

Another aspect of this problem involves "branching." The 
Congressional Research Service estimates that one-third of all 
proprietary schools are branch campuses. When institutions 
expand rapidly by setting up branches in distant parts of the 
country often in totally different occupational fields the 
ability to administer the main campus effectively as well as 
remote branches is stretched thin. Of particular concern is the 
ability of a school to administer the student aid program, where 
the rules are both numerous and frequently changing. Serious 
deficiencies among branch campuses may be obscured because (1) 
separate accreditation of branch campuses is not required if the 
main campus is already accredited; and (2) default rates and 
other performance measures may reflect scores averaged across 
branch and main campuses. 



The federal government has a special responsibility to make 
sure that schools participating in the student aid program do not 



engage in financial abuse and do not go out of business leaving 
indebted students without an education. ED is responsible for 
ensuring that schools meet minimal financial standards and that 
they manage the substantial federal resources they receive 
appropriately . 

The establishment of strict financial standards is appropriate 
where the objectives of a performance-based system are to set 
minimum standards for eligibility, to target program monitoring 
to concentrate on high-risk institutions, and to protect 
consumers from abuse. 

Current Practice . ED employs financial and administrative 
criteria when initially certifying schools to participate in 
student aid, although serious questions have been raised about 
the rigor with which existing requirements are enforced. The 
financial criteria on which the Department relies is essentially 
a measure of net worth. Companies must demonstrate a positive 
net worth on a financial statement. Schools are required to 
submit additional financial statements every two years. These 
statements focus exclusively on the expenditure of student aid 
funds. Schools that are considered financially deficient, may be 
required to post a surety bond. Various proposals have been made 
to require more frequent financial data and to increase data 



14. U.S. Senate, "Abuses in Federal Student Aid Programs," pp. 24-28. 
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reliability by requiring audits by a Certified Public Accountant 

(CPA) or independent certification of the results. 

The problems associated with financial certification have been 

extensively reviewed by others Among the limitations cited 

are the following: 

o Little is done to evaluate the financial information 
provided ; 

o Required reports are frequently submitted late or not at 
all; 

o There are no requirements for recertif ication, with the 
exception of schools that change owners; 

o Surety bonds are insufficient to protect students and to 
compensate the federal government for losses; 

o Program reviews fail to detect financially troubled 
institutions; and 

o There is no mechanism to verify independently the 

information provided by applicants seeking certification. 

Administrative capability also is considered in certifying 
schools for student aid. Two indicators that ED established to 
indicate administrative capability are (1) student withdrawal 
rates in excess of 33 percent; and (2) adequacy of staffing to 
administer student aid. With respect to withdrawal rate, the 
Inspector General found that "institutions are not denied 
certification no matter how far they exceed these tolerance 
levels." other "administrative capability requirements" (e.g., 



15 See Margot Schenet, 'Proprietary Schools: The Regulatory Structure" Congressional Research Service, (Washington. DC: 
90.4240 EPW, 1990); Charles Brennan. "Issues Noted to Date in Our Audits of the Postsecondary Accreditation, and Institutional 
Eligibility and Certification Processes,' (Washington, DC: U.S. Inspector General, 1990). 
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acknowledgment of having received and understood the student aid 
rules) are largely paperwork 

ED also employs several fiscal indicators, plus other 
information, in deciding whether a school's program should be 
reviewed. These indicators are: the school has a default rate 
between 30 and 40 percent the school has large audit 
liabilities or overdue audit reports, there has been no program 
review within four years, the school is new, the school belongs 
to a chain with multistate locations, the school is being 
monitored for financial capability, and the volume of GSL and 
Pell grants has increased. 

One major problem with the indicators currently used to 
identify financially risky institutions is the inadequacy of the 
available data. ED's data base theoretically contains data on 
most pertinent factors associated with financial risk, but the 
data are often incomplete or out of date. Data on GSL and Pell 
volume are at least 18 months old before they enter the 
Department's data base; substantial losses may occur before any 
data with which to detect potential abuse become available. 
There are no data on the percentage of students who fit the 



16. In addition to establishing the financial and administrative criteria used in determining initial eligibility for student aid. 
ED determines whether a school has received a state license to ofX'rate and is accredited by a recognized accrediting agency. 
Studies by the Inspector General and the Congressional Research Service have suggested that state licensing and accreditation 
practices are often superficial. 

17. Schools with default rates exceeding 40 p *r:ent are to be reviewed by guarantee agencies. 
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category of having the "ability to benefit" or the percentage of 
revenue spent on instruction versus recruiting. Other key data 
(branching, net worth, percentage of students on federal 
financial aid, bonding, arrangements in event of school failure) 
are not regularly updated. 

Another significant problem is that the financial indicators 
employed have not been scientifically validated through an 
independent random sample of institutions. ED identifies risk 
factors and assigns weights to those factors entirely on the 
basis of staff exptirience. In the absence of an analysis of 
financial abuse factors based on a random sample, there is no way 
to determine whether program reviews are being optimally 
targeted, or whether a different configuration of factors and 
weights would identify other high-risk schools. 

Alternative Indicator Systems to Iden tify Financial Stability and 
Student Aid Abuse . Although studies have documented a problem of 
lax or inconsistent enforcement of existing rules and 
procer'ures^^ and much more could probably be done with the 
financial and administrative information currently available, 
several additional financial risk factors could be employed (see 
Table 2) in decisions about recertif ication or program reviews: 
o Percentage of students receiving federal financial aid; 



18. U.S. Inspector General, "Financial Analysis Certification Process Not Adequate to Protect Students and Government.- 
Audit Control Number 11-80160, 1989. 
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Measures 



Financial solvency 



Potential abuse of 
student aid 



to 



37 



Table 2 
Financial Risk Indicators 

Availability 
Data Needs of Data 

Net worth Yes 
Adequacy of bonding Yes 

Financial protection No 

for students against 

insolvency 
Percentage of students on Yes 

federal financial aid 
Large increases in Yes 

volume of GSLs and 

Pell grants 
Large increases in Yes 

defaults 
High default rate Yes 

Change in school Yes 

owner£;hip 
Extensive branching Yes 
Adequacy of fin icial No 

arrangement for student 

if school fails 
Changes in tuition Yes 

consistent with maximum 

available student aid 
Percentage of abililty- No 

to-benefit students 
Percentage of revenues No 

derived from federal 

aid 

Percentage of revenues No 
spent on instruction 
and recruitment 



Use 

Currently used for 
certification and 
recertification 



Not used 
Program reviews 

Not used 

Eligibility, program 

reviews 

Not used 

Program reviews 
Not used 

Not used 

Not used 
Not used 

Not used 
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o Percentage of a school's revenues derived from 
financial aid; 

o Percentage of revenues spent on recruiting and on 
instruction; 

o significant increases in defaults; 
o Changes in school ownership; 

o Adequacy of financial arrangements to protect students 

in the event of school failure; and 
o Changes in the relationship between price charged and 

maximum available aid. 



I With large sums of money, many vendors, and complex 

I transactions to consider, the task of reviewing all financial 

* records to identify financial abuse considerably exceeds current 

I capabilities. This problem is common to other large federal 

programs that use many service suppliers. Food Stamps, Medicare, 
I the Supplemental Feeding Program for Women, Infants and Children 

(WIC) , and the Internal Revenue Service have faced similar needs 
■ for systems to monitor and detect potential error or financial 

I abuse. They have developed high risk profiles to identify 

sources of potential abuse. Applied to ED, such systems would 
I simplify the review of wide-ranging financial data, identify the 

schools where the risk of abuse is greatest, reduce the time 
I before potential abuse is identified, and target the Department's 

I very limited investigatory capabilities on these institutions. 

I 
I 



All the federal programs just mentioned have developed "risk 
profiles" to identity the greatest sources of potential abuse. 
Their systems employ statistically validated "risk factors" to 
predict financial error and abuse. A similar system could be 
developed for student aid using many of the indicators noted 
earlier to detect potential financial abuse."" A parallel 
system could be developed to predict potential proprietary school 
failures. 20 



Risk factors that make up a system to detect error may be 
identified on two bases: 

o The correlation between potential risk factors and 

actual abuse in a random sample of institutions. Data 
from a random sample of proprietary schools do not now 
exist but could be incorporated in future program 
reviews . 

o Experiential knowl^ficrp about the co rrelates of abuse 
based on program reviews that the Department has 
conducted . In the short run, a risk profile based on 
program experience could be generated, but statistically 
valid adjustments to such a system will ultimately 
require data from a random sample of institutions. 



Any system to identify potential financial abuse needs an 
investigatory capability and effective sanctions. Risk 
indicators are simply correlates of financial abuse, not direct 
evidence. By themselves, they are inadequate to justify 



19. ED has developed a computer based risk profile system to Identify applications for student aid that are most likely to 
contain error; such high-risk applicants are then targeted for verification. 

20. New York State has developed such a system. As of this writing, we have been unable to determirie how effective that 
system has been. 

28 
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withholding a school's certification for student aid. Financial 
risk factors permit efficient targeting of investigations and, if 
the system works well, detection and deterrence of abuse before 
substantial losses can occur. Currently, ED's -ability to 
investigate high-risk schools effectively and to impose sanctions 
that stick is highly suspect. Investment in the development of 
better me^isures and indicators to detect potential financial 
abuse should be undertaken only if parallel developments occur in 

• 21 

the ability to investigate schools and invoke sanctions. 

In summary, although ED currently collects certain types of 
information pertinent to assessing financial risk, serious 
questions have been raised about the timeliness, completeness, 
and use of these data. The Department should collect additional 
indicators of the sort already described and strive to improve 
the timeliness of data that it obtains. Financial analysts, 
accountants, and bankers should be asked to help identify better 
financial indicators. Better measures and indicators, however, 
must be accompanied by improved investigations. 

Indicators of Program Inputs 

The search for educational quality has led researchers to seek 
special configurations of resource allocation, faculty 
characteristics, curriculum, teaching style, organizational 



21. A discussion of the requirenents for a better system of on-site investigation of potential abuse and of credible sanctions 
is beyond the scope of this report. 
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climate, and leadership traits that are consistently more 
effective than other arrangements. This search has led to 
efforts to identify "effective schools" and to replicate 
educationally effective practices in additional settings. In 
recent years, the effective schools reform movement appears to 
have lost ground as an instrument of educational policy, in favor 
of an emphasis on accountability based on student outcomes. 

The recently completed National Assessment of Vocational 
Education (NAVE) attempted to identify effective practices in 
postirecondary vocational education through a study comparing 
training programs that were highly rated with programs that were 
less well rated in both the public and private sectors. This 
study found that five main factors related to high program 
ratings (by employers) 

1. Intensity and depth of vocational instruction? 

2. Integration of theoretical and applied aspects of 
vocational instruction; 

3. Frequency of external linkages through advisory panels 
of employers and customized training programs, as well 
as other means that keep faculty abreast of developments 
in their field; 

4. Quality of the job placement function; and 

5. Dissemination of "honest" program information. 



22. Rocoo DiPietfo, Lou Tornatzky, James Jacobs, and Michael Woods, "Predictors of High Quality Postsecondary Vocational 
Education," NAVE Contractor Report (Ann Arbor, Ml: Industrial Technology Institute, February 1989). 

23. David Goodwin, Rnal Report, National Assessment of Vocational Education: Postsecondary Vocational Education. Vol. IV 
(Washington, D.C.: U.S. Government Printing Office, 1989), pp. 27-36. 
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The factors identified in the NAVE study were quite general 
and frequently difficult to measure objectively. Many other 
factors that were expected, on the basis of educational and 
organizational theory, to be related to program effectiveness— 
the quality and quantity of equipment available, faculty 
educational background, and various attributes of school leaders- 
were not found to be significant in this study. Although this 
attempt to identify characteristics of effective vocational 
training programs produced interesting results, the generally 
weak findings provide an inappropriate basis for setting policy 
standards . 

Fundamental beliefs about key attributes of program quality 
are occasionally challenged on the basis of new research 
findings. A recent Rockefeller Foundation study of minority 
single mothers found that "by offering in-depth training in 
occupational skills with strong support services, poor single 
mothers can quickly increase their earnings and employment 

rate Programs that emphasize basic skills education rather 

than immediate skills training do not improve earnings as 
quickly."^* This finding is contrary to the well-established 
view guiding much federal policy that effective job training 
programs rest heavily on a foundation of basic skills. Thus, new 



24. The Rockefeller Foundation, "Into the Working World: Research Rndings from the Minority Female Single Parent 
Program." 



approaches to learning rest heavily on teaching skills in an 
applied context. 

Proprietary school accreditation agencies consider a wide 
range of educational inputs and processer when evaluating a 
school: curriculum content, student recruitment and admissions 
policies, instructional staff, space and facilities, 
instructional materials and methods, and student support 
services. For example, the Accrediting Commission of AICS 
(Association of Independent Colleges and Schools) has a standard 
that requires faculty preparation that is "academically and 
experientially appropriate to the subject matter taught. Faculty 
members should be competent to teach the subject matter offered 
and should have reasonable latitude in their choice of teaching 
methods". 

In addition to faculty credentials and instructional 
procedures, accreditation agencies may require that program 
content be carefully planned and appropriate to the occupational 
objective, that student progress be continuously assessed and 
documented, and that information disclosed to the public not be 
misleading. 



25. Secretary's Commission on Achieving Necessary Sl<ills; US Department of Ubor, "What Work Requires of Schools: A 
SCANS Report for America 2000;" June 1991. 

26. The Accrediting Commission of the Association of Indapendent Colleges and Schools, "Accreditation Criteria," 1989, p. 

47. 
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Review teams that visit schools consider all of these factors 
when information is gathered through on-site observation. There 
are no absolute standards. Assessment and ultimately, 
recommendations about whether to accredit a school reflect the 
subjective judgments of team members. This approach differs 
markedly from the requirement that an indicator system employ 
quantitative measures that permit valid comparisons across 
institutions. 

State laws and rules governing the licensing of proprietary 
schools may also focus on a variety of educational inputs 
including space and facilities, faculty qualifications, and 
accurate public disclosure of claims to the public. The Nunn 
Committee reported that licensure requirements vary widely across 
states, from merely filling out a form for a general business 
license to undergoing a more thorough evaluation by a state board 
or agency. In some states, schools are automatically licensed if 
they have been accredited. 

The effective schools movement, findings from the National 
Assessment of Vocational Education, and state licensing and 
accreditation agency practices all demonstrate the difficulty of 
establishing statistically valid predictors of program quality 
that are based on educational inputs. Although educators 
generally agree on many aspects of what constitutes program 

27. U.S. Senate, "Abuses in Federal Student Aid Programs," p. 14. 



quality, these notions are based largely cn experience and 
intuition. 

Key indicators such as "faculty qualifications," 
"curriculum content," and "adequacy of facilities" are unlikely 
ever to be defined with enough precision to permit ^quantitative 
measurement. In this case, it appears best to rely on the 
judgment of qualified outside reviewers and to ensure that such 
reviews are rigorous, independent, and thorough. Even if 
meaningful standards for educational inputs could be established, 
however, it is questionable that these factors represent the best 
way to measure program quality. Increasingly, educational policy 
has focused on school outcomes — what students have learned — as a 
measure of program effectiveness, rather than on school 
characteristics . 

Indicators of Program Outcomes 

There is a substantial and growing effort to incorporate 
outcome measurement and performance incentives into most federal 
education, job training, and social welfare programs. Similar 
efforts are under way in many states. These efforts rest on the 
premise that replacing the current focus on rules to monitor 



28. The establishment of formal federal standards related to program inputs and content, it could also be argued, represents 
an unwarranted step toward government control over school curriculum. 



compliance with procedures in favor of an emphasis on student 
outcomes will create incentives to improve results for students. 

Improving the opportunity for students to acquire labor market 
skills, obtain jobs, and improve their economic status are the 
key objectives of any vocational training program. Performance 
measures oriented toward these outcomes offer an opportunity to 
hold schools accountable for results that matter most. An 
effective system of accountability based on educational outcomes 
could complement, and in some cases replace the current focus on 
school inputs, defaults, and financial risk. 

JTPA's performance standards system has generally been 
credited with increasing the emphasis on placements and 
preemployment competencies and with creating a strong sense of 
accountability for the use of public funds. ^' In JTPA, up to 6 
percent of state funds are set aside as an incentive to reward 
service delivery agencies (SDAs) that perform well. In turn, 
many SDAs enter performance contracts with organizations that 
svipply training; under these contracts, a share of funds paid is 
contingent on student outcomes (e.g., 62 percent follow up 
placv3ment rate) . The Job Opportunities and Basic Skills (JOBS) 
program in the Family Support Act also plans to institute outcome 
measures to assess performance. 



29. National Commission for Employment Policy, "JTPA Performance Standards: Effects on Clients, Services, and Costs". 
1988. 
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The recently reauthorized Carl Perkins Vocational and Applied 
Technology Education Act mandates that states develop systems of 
performance measurement and standards. The Perkins Act requires 
that states measure learning and competency gains in academic 
skills and at least one more of the following: competency 
attainment, job or work skill attainment, retention in school, 
and job placement. Under the Perkins Act, states have complete 
discretion in defining these indicators, specifying measures, and 
setting standards. There is no requirement that funding or other 
rewards and sanctions be tied to performance outcomes. 

Should student aid move toward performance standards, one key 
issue is the extent to which federal guidelines, measures, or 
standards should be employed. Another issue is how to make 
performance standards and measures for public programs receiving 
federal Vocational Education Act funds compatible with standards 
for other vocational education programs, largely proprietary, 
that are financed with federal student aid funds. 

State Practices in Measuring Postsecondarv V ocational 
Outcomes. A recent Department of Education study examined the 
extent to which states collect information from postsecondary 
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vocational programs cn a wide range of student outcomes, with 
the following findin'js: 

o Most states collect some information about program 

outcomes (see Table 3). The most frequently collected 
outcome measure is the number of students who complete a 
training program (found in 46 states for public schools 
and 29 states for proprietary schools) . The next most 
frequently collected measure is the extent to which 
program completers are placed in jobs (33 states for 
public schools and 21 states for proprietary schools) . 
Most states attempt to determine whether the job 
placement is related to the training provided, although 
this practice is much less common for proprietary 
schools than it is for public schools. A few states 
collect information on outcomes such as earnings, gains 
in basic skills and job knowledge, and results from 
state licensing exams. 

o Most states that measure completions have developed 

standardized definitions of who is a "program completer" 
(39 of 46 states in the public sector and 26 of 29 
states for proprietary schools) . In most cases, 
completers are defined as recipients of formal degrees. 
One critical limitation to the definition of program 
completer used for public institutions is that 
information is c xlected only for recipients of 
associate degrees. 



Table 3 

Number of States Collecting Student 
Outcome Information on Postsecondary Vocational Programs 

Spring 1990 

From from 

outcome Measure Community College Proprietary Schools 

Completion 46 29 

Placements 33 21 

Earnings 21 3 

Gains in basic skills 13 0 

Licensure 12 2 

Source: Research and Evaluation Associates, "State-Level 

Measurement of Postsecondary Vocational Education" 
(Washington, D.C. 1991). 



30. "State-Level Measurement of Performance Outcomes in Postsecondary Vocational Education, Vol. I, Executive Summary 
and Overview of State Policies," prepared for U.S. ED by Research and Evaluation Associates. Washington, D.C, 1991. 
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o The most common source of information about job 

placement is surveys of former students; the second most 
popular method is surveys of employers. 

o Few states attempt to verify the accuracy of data reported 
'ly schools. 

o No state measures program outcomes the same way for 
students in public postsecondary programs and for those 
in proprietary school prorrams. This difference 
reflects the division of responsibility for 
postsecondary vocational training among a variety of 
state agencies, including those responsible for 
community colleges, public vocational technical 
institutes, and multiple licensing boards for different 
occupational areas. 

Alternative Measurer . In the context of student aid, the 
three types of student outcomes that are most relevant are labor 
market outcomes, learning outcomes, and program completion 
outcomes . 



o Labor market outcomes include the rates at which students 
are placed in jobs^ the job retention rate, the extent to 
which job placements are in a field "related to the 
training," the level of earnings at job entry and selected 
times thereafter, and employer satisfaction with the 
school's graduates. 

o Learning outcomes include the rates at which students 
demonstrate occupational knowledge and competency by 
passing state licensing/certification exams and by 
achieving certain scores on occupational competency exams. 
Learning outcomes may also include measures of academic 
skills, growth in academic skills (as required by the 
Perkins Act), and measures of student satisfaction. 

o Program completion outcomes include dropout and completion 
rates, rates at which students obtain more advanced 
training (on the job or in school) , the ratio between 
completion rates for student aid recipients and other 
students, and the ratio between completion rates for ATB 
students and non-ATB students. 
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Each of these types of performance outcomes — labor market, 
learning, and program completion — is discussed in the sections 
that follow. 

Labor Market Outcomes . Given that the principal purpose of 
occupational training in proprietary schools is to prepare 
students for jobs, labor market indicators must play an important 
role in assessing program performance. In JTPA, employment and 
weekly earnings at follow-up 13 weeks after program c pletion 
are two of the basic indicators by which client outcomes are 
measured. These are indicators of job retention and earnings 
growth. In the past, JTPA performance indicators placed more 
emphasis on initial job entry. 

Tennessee uses job-related placement as one of the outcome 
measures to assess its two-year public colleges. The placement 
rate is calculated as the ratio of students placed in fields 
related to their training to the total number of program 
graduates— minus those serving in the military or pursuing 
further education.'^ institutions are required to survey their 
graduates and to maintain auditable records. The state maintains 
a list of occupations that it deems to be related to the program 
of study. Florida requires that public schools attain a 70 
percent job placement rate to receive state vocational education 



31. State of Tennessee, 'Performance Funding Standards For Public Colleges and Universities." revised February 15. 1 



funds, and California has enacted, although not yet implemented, 
a similar job placement standard for proprietary schools. 

To identify schools that recruit students simply to obtain 
financial aid, placement rates for aided students can be compared 
with the rates for unaided students (see Table 4) . Rates for 
students with financial aid that are substantially lower than the 
rates for unaided students may indicate a pattern of abuse. A 
similar analysis could be made to compare placement rates for ATB 
students with the rates for high school graduates. (Similar 
comparisons between aided and unaided students, ATB and non-ATB 
students should be made for indicators of occupational competency 
and program completion.) 

Some states survey employers and students directly to 
determine job placement rates and assess satisfaction with 
graduates of vocational programs. A recent ED study found nine 
states currently conduct some form of employer survey. Most 
state surveys focus on graduates of public vocational programs. 
Finally, as requested by Congress, the National Center for 
Education Statistics is currently conducting a feasibility study 
to determine whether job placement rates can be reported by 
institutions under the Student Right to Know Act. This report is 
scheduled for submission to Congress by August 1991; its 



32. Research and Evaluation Associates, "State Level Use of Employer Surveys to Assess Performance Outcomes in 
Postsecondary Vocational Programs," Draft, July 1991, prepared for U.S. Department of Education under contract EALC890010. 
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Table 4 

Labor Market Outcome Indicators 



Measures 
Job placement rate 

Job retention rate 



Training-related 
placement 

Initial earnings 



Earnings growth 



Employer satisfaction 

Ratio between place- 
ment rate for aided 
students and non- 
aided students 

Ratio between place- 
ment rate for ATB 
students and all 
other students 



Availability 
of Data 

No 



No 

NO 

NO 

NO 

No 
NO 



No 



Sources of Data 

Student surveys, 
state wage records 

Student surveys, 
state wage recorris 

Student/ employer 
surveys 

Student surveys, 
state wage records 

Student surveys, 
state wage records 

Employer surveys 

Student surveys, 
state wage records 



Student surveys, 
state wage records 



Uses 

Not currently used. 
Could be used for 
minimum standards, 
performance contracts, 
rewards for excellence, 
consumer information 
and system assessment. 



conclusions will be based on all types of postsecondary 
institutions, not exclusively proprietary schools. 

Using different approaches, several recent studies have 
estimated various labor market outcomes for students enrolled in 
postsecondary vocational education. With respect to proprietary 
school students, the National Assessment of Vocational Education 
found that about 26 percent of proprietary school students 
experienced unemployment during a 12-month period following 
training and that proprietary school students were placed in jobs 
related to their training about 66 percent of the time, for which 
they received an average wage of $8.59/hour.^^ The 
congressional Research Service (CRS) estimated employment rates 
of 77 percent for men graduating from proprietary schools and 74 
percent for women. CRS found that male proprietary school 
gradua.-es earned $7.85 per hour and females earned $6.47 per 
hour.^ A study by Norton Grubb that looked at long term 
economic outcomes found average hourly wages of $8.93 for men 
with certificates from proprietary schools and $7.22 for women 
with similar credent ials.^^ should be noted that all these 

findings are averages for the entire population of students, and 
not for specific institutions. 

33. Goodwin, "Post*{*condary Vocational Education," Vol. IV, pp. 53-74. 

34. Rotsert Lyke and Thomas Gaije. "Early Labor Market Experiences of Proprietary School Graduates" (Washington. D.C.: 
Congressional Research Service, 1991). 

35. Norton Grubb, "Long Run Effects of Proprietary Schools on Wages and Earnings: Implications for Federal Policy," 1991, 
unpublished. Table 6. 



The principal difficulty in measuring labor market outcomes on 
an institutional basis — as is required in a performance based 
system-is the inadequacy of data. Performance based systems that 
involve high stakes— where eligibility fcr student aid may be 
denied to schools on the basis of student performance— require a 
high level of confidence in the accuracy of the data. Safeguards 
need to be established to prevent distortion or lying in the 
reporting of data. There are other limitations as well. With 
respect to labor market outcomes, school- administered follow-ups 
of graduates are frequently characterized by high levels of 
nonresponse, and lack of objectivity by the school in determining 
when students are employed (or unemployed) or have been placed in 
fields related to their training. 

Several strategies could be used to address these limitations: 

o Validate school generated surv'^ev r ^^sult^s by selecting a 
T-anrinm samplQ of graduates f or independ e nt follow-up . 
Several proprietary school accrediting associations 
apparently estimate placement rates and assess employer 
satisfaction by independently contacting a random sample 
of students and their employers. 

o Make more extensivp- ugp- of state nnemplovment wage 

record data to me as yre emplovroent and earnings. Because 
such information is provided by employers, it is 
objective, not subject to student recall, and can be 
used to meai^ure both entry-level and long-term 
employment and wage rates. 

ED currently has a study in progress to assess the feasibility 
of using state unemployment and wage record data to obtain 
information on earnings and employment of vocational program 
graduates. Preliminary evidence suggests that wage records, when 
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used with other existing data systems (federal employment, 

military, higher education enrollment) , can be expected to 

provide labor market outcome data on from 60 to 80 percent of the 

students.^ This "hit rate" is substantially higher than the 

norm for school alumni surveys. Several other efforts to use the 

wage record data system are under way in JTPA, in the Department 

of Education, and in at least a half-dozen states. In 

reauthorizing the Perkins Vocational Education Act, Congress 

mandated additional studies of this resource by the National 

Occupational Information coordinating Committee (NOICC) and the 

Office of Technology Assessment. 

o Develop objective rules for determinin g when a job 
placement is "related" to a student 's training. The 
National Assessment of Vocational Education built on 
work begun by (NOIC) to link fields of training to 
different occupational classifications. Tennessee has 
taken a similar approach in determining job related 
placements. More work needs to be done in developing 
such a "cross-walk" between training and jobs. 

o Where school surveys of graduates are used to determine 
job placement, raise the notoriouslv low response rates 
by 1) requiring a minimum response rate, with all 
nonrespondents above the minimum counted as 
nonplacements; or 2) treating all nonrespondents as 
graduates who did not get placed in a job.^ The 
latter approach is used in Tennessee. 



Learning Outcomes . Measuring learning outcomes is far more 
complicated than measuring labor market outcomes. Methods of 



36. In Rorida, where the state has had considerable experience tracking students with the use of state wage records and 
other existing data sources, ♦he "hit rate" is 75 to 80 percent for community college and university graduates. Other studies 
have obtained even higher hit rates. 

37. JTPA requires a 70 percent rate and assumes non-respondents are distributed in the same proportion of employed and 
unemployed as respondents. 
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testing both general and occupationally specific knowledge and 
skills or competencies are available in a growing number of 
fields, but there are problems with the number of fields for 
which recognized tests currently exist and the adequacy of test 
content. Despite these problems, measures of general and specific 
occupational competency have the potential to provide a uniform 
measure of learning outcomes that is comparable across similar 
programs. In contrast, job placements may be affected by 
external factors such as local labor market conditions, student 
nonresponse, or school manipulation of the data. 

There are three main ways (see Table 5) of assessing 
occupational knowledge and competency (see Appendix B for a more 
detailed discussion of occupational competency testing): (1) pass 
rates on state licensure exams; (2) tests developed by 
independent sources; and (3) tests developed by the programs 
themselves . 

o Pass rates on state licensure exams . Not all fields are 
licensed, and licensing practices and standards may vary 
across states. Fields that are typically subject to 
licensing exams include cosmetology, health care, truck 
driving, electrical repair, certain trades, and real 
estate. 

Fields that are not subject to state licensing exams include 
secretarial work, computer data processing, marketing, drafting, 
accounting (below the CPA level), and machine tool occupations. 
Fields not subject to state licensing represent more than 50 
percent of total proprietary school enrollment. In fields where 
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Table 5 



Learning Outcome Indicators 



Measures 

State licensing/ 
certification 
exams 

Nationally recognized 
tests of occupational 
competency 

School -developed 
test of occupational 
competency 



Availability 
of Data 

No 



No 



No 



Sources of Data 
States 

Schools 

Schools 



Uses 

Not used. Could be used 
to set standards of school 
eligibility for student 
aid, performance contracts, 
rewards for program improve- 
ment, provision of consumer 
information and system 
assessment. 



state licensing exams are administered to program graduates, 



however, the rate at which students pass these exams is an 

appropriate measure of occupational competency. 

o Tests developed by independent sources . Current sources 
of occupational competency exams include the National 
Occupational Competency Testing Institute (NOCTI) , 
the Educational Testing Service (ETS) , the Instructional 
Materials Laboratory at Ohio State University, the 
Vocational Technical Consortium of States (V-TECS) , and 
the American Association for Vocational Instructional 
Materials (AAVIM) in Athens, Georgia. ^° 



NOCTI is a private organization that has developed 
occupational competency tests in about 70 fields corresponding to 
programs offered in proprietary and public vocational schools. 
Mrst of these tests, which include written and performance 
components, cost less than $7 to administer. Other test vendors 
include the ETS, which has developed a limited number of 
occupational tests. V-TECS has developed a test "item bank" that 
states can use to develop tests tailored to a particular 
curriculum or local community. 
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Several states, including Colorado, Wisconsin, Michigan, and 
Florida, maintain centralized, state-level, test development 
agencies. These agencies develop tests used in fields subject to 
state licensing and certification. In other states, state 



38. Some proprietary schools have said that states will report test results only to individual students, and not to the school. 
This problem would have to be resolved. 

39. NOCTI has developed tests in approximately 60 different fields covering secondary and postseccndary training. 

AO A study by the Office of Technology Assessment found, as of 1989, 13 states engaged in testing the occupational 
competencies of vocational and technical students, and 7 states involved in developing competency tests for vocational 
students. 
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departments of education have developed occupational tests that 
may be used as exit exams or for diagnostic purposes before 
students take a board certification test. In Oklahoma, such 
tests cover 55 to 60 percent of the occupational trades taught in 
state community colleges and public vocational schools. 

Finally, various professional organizations have developed 
exams that may be used by states. Such exams exist in welding, 
cosmetology, and nursing. The National Council of State Boards 
of Cosmetology has developed tests for barbers (used in whole or 
part by 21 states), manicurists (26 states), general cosmetology 
workers (36 states), aestheticians, electrologists, and several 
other specialties. 

o Tests developed bv the program s themselves. 

Tennessee requires that institutions demonstrate that 
graduates are competent in their major field and that overall 
test scores improve over time. Where state licensing exams are 
not offered, several schools offering similar training must adopt 
or develop a common test. Schools may also develop their own 
tests. All tests developed are required to be validated by 
independent content experts and a psychometric ian.^^ 



41. Tennessee Performance Funding Standards. 
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Although researchers are working to identify occupational 
competencies and to measure the skills that students possess, 
much more work needs to be done to ensure that tests exist in all 
the fields tl ey are needed and that these tests are of high 
quality. No systematic survey of available tests has recently 
been undertaken, but the best guess is that nationally recognized 
and normed tests of occupational competency do not now exist in 
many of the fields where proprietary schools offer training. 
Where content -valid occupational competency tests exist, they 
have been found to be highly valid predictors of job 
performance.*^ 

Before any federal requirement that students be tested to 
assess their occupational competency is instituted, however, the 
range of available tests should be more systematically 
investigated. 

In the meantime, a phased-in approach to assesoinent could 
begin with the fields where state licensiiig exams are 
administered or where othvsr recognized tests of occupational 
competency have been developed. Pending further development, 
gaps could be filled with cooperatively developed or 
institutionally developed tests (the Tennessee approach) . 



42. John Bishop, "Occupational Competency as a Predictor of Ubor Marl<et Performance," unpublished. Cornell University. 
NYSSILR, 1988. 



A second problem concerns test content. In response to 
proposals for more extensive and different ways of testing 
student learning, educational testing is undergoing many changes. 
Paper and pencil tests of occupational knowledge, the predominant 
approach in many occupational fields, may be replaced by 
alternative measures that gauge a wider variety of work relevant 
skills in a "hands on" manner. Competency tests developed by 
NOCTI and Oklahoma offer hands-on performance tests as one 
component of the testing program. 

The report of the Department of Labor's Secretary's Commission 
on Achieving Necessary Skills (SCANS) , suggests new approaches to 
measure learning for work-bound students. Whereas what students 
learn may ultimately be the best indicator of whether they have 
benefitcid from a job training program, the measurement of 
learning is difficult, and test quality is not uniform and may 
not be very high. 

There is an important role for the federal government in 
encouraging the development of high-quality tests. Federal job 
training and vocational education programs — JTPA, JOBS, 
Vocational Education, and, in the future, perhaps student aid- 
will all have systems of performance standards in which the 
assessment of student learning is vital. An interdepartmental 
effort among the Departments of Education, Labor, and Health and 
Human Services to spur development of high quality occupational 
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coiupetency tests may be justified. This should be undertaken as 
a long-term (five-year) effort. 



Program Completions . The program completion rate, a key 
component of student attainment, is the ratio between the number 
of students who enter a program and the number who actually 
graduate. Because students may enroll briefly and quickly drop 
out before incurring any debt, an alternative definition of a 
completion rate may be limited to students enrolled long enough 
to have a student loan. Completion rates may be calculated to 
represent a simple average for a recent time period or to 
indicate changes (improvement or decline) for an institution. 

In addition to the overall completion rate for all students, 
completions for particular subgroups may be of particular 
interest in the context of student aid policy (see Table 6) . Two 
groups of special concern are students who receive federal 
financial aid and "ability to benefit" (ATB) students. 
Meaningful measures of performance are to compare completion 
rates for aided and unaided students and for ATB and non-ATB 
students. These measures indicate the extent to which the 
benefits that aided students and ATB students get from their 
education are similar to the benefits that unaided and non-ATB 
students get. 
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Table 6 
Program completion Outcomes 



Availability 

Measures of Data Sources o f Data 

Overall completion Required under School 
rate Student Right 

to Know Act, as 
of 1993 



Completion rate 
change over time 



Required under 
Student Right 
to Know Act, as 
of 1994 



Schoul 



Completion rate 
for recipients of 
federal student aid 



No 



School 



Ratio between No School 

completion rates 
for aided students 
and nonaided 
students 

Completion rate for No School 

ATB students 

Ratio between No School 

completion rates 
for ATB students 
and all others 



Uses 



currently, withdrawal 
rates above 33 percent 
are considered an 
indicator of 
"administrative 
impairment" in 
certifying schools 
for student aid. 

Not used 



Not used 



Not used 



Not used 



Not used 



Using nationally rapresentative survey data, NAVE found 
completion rates of about 36 percent (within four years after 
high school graduation) among proprietary school students/^ 

Completion rates are important, for studies have found that 
students who complete their training are much more likely to 
derive economic benefits than are students who do not/'^ 
According to the Congressional Research Source (CRS) , "On 
average, men and women who completed proprietary schools had 
higher hourly earnings than those of their gender who only 

attended high school For both men and women, no significant 

differences in hourly earnings were found between those who 
attended but did not complete proprietary schools (or any other 
schools) and those who only graduated from high school." 

Completion rates will be available July 1, 1993, from data 
that schools are required to make available to students under the 
Student Right to Know Act/^ Feasibility studies are also under 
way to assess the collection of more detailed data on the 
completion rates for various sub-populations, on the rates at 



43 Goodwin Postsecondaiv Vocational Education , p. &-2. Although the completion rate for proprietary school students may 
seem low this rate is consistent with the rates for students in community colleges and public vocational schools. In contrast to 
community colleges, however, where most completers earn associate degrees, the majority of proprietary school completers 
earn certificates. 

44. Ibid., pp. 70-71. 

45. Robert Lyke and Thomas Gabe, "Early Labor Market Experiences of Proprietary School Students" (Washington, D.C.: 
Congressional Research Sewice, 1991), pp. 25-26. 

46. Schools are not required to report completion rates to the Department but this could be mandated. 
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which individuals pass licensure and certification examinations, 
and on job placement rates. Although these feasibility studies 
do not include completion rates for students who receive federal 
aid and students who did not, or for ATB and non-ATB students, 
completion rate data could certainly be expanded to provide 
brealcdowns among these categories. Thus completion rate data can 
be obtained at little additional cost by piggybacking on 
requirements in the Student Right To Know Act. 

Mnn-ipio Rtnrip pi- outcome Measures . Occupational training has 
multiple goals. No single measure of student outcomes is likely 
to be an adequate indicator of program performance. Moreover, 
use of a single measure may result in perverse effects, as 
schools attempt to maximize their performance ratings. Exclusive 
use of occupational competency measures could encourage narrow 
"teaching to the tp^st," job placement measures could promote the 
substitution of job search assistance or placement in low-quality 
jobs for more fundamental occupational training, and degree 
completion measures could result in a dilution of graduation 
standards . 

If schools can be rewarded for doing well in one area (e.g., 
placement) but can ignore their responsibilities in other areas 
(student learning) , student outcomes as a measure of program 




performance may create undesirable incentives. Therefore, it is 
important that any federal policy in this area provide for 
multiple definitions of performance. 

"Value-Added" and Improvement Measures . Performance outcome 
systems have a variety of objectives: to set minimum standards, 
to provide information to prospective students, and to encourage 
program improvement. Wliere program improvement is the goal, it 
is important that all institutions— not just the lowest 
performing (who wish to exceed minimum standards) or the highest 
performing (who may wish to compete for rewards) —have incentives 
to improve their performance. To ensure that all institutions 
are encouraged to improve, assessments of institutional 
improvement and "value added" should be incorporated in any 
measurement system. 

Measures of institutional improvement reflect changes over 
time. All three types of student outcomes — labor market, 
learning, and program attainment— may be addressed as measures of 
institutional improvement. Value-added measures focus on average 
learning gains (differences between test scores for students 
before and after their training) , rather than on gross levels of 
knowledge and skills. Value-added measures ^re one way to 
compare the amount of learning at schools serving students of 
different ability or educational background. 
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Where minimum standards of school eligibility for student aid 
are established, useful institutional comparison on the basis of 
student outco*Ties requires an adjustment for student 
characteristics (poverty, age, academic ability, high school 
graduation status) , length of program, and local labor market 
conditions. These adjustments are important, for completion 
rates in programs that are one or two years long cannot be 
expected to be the same as completion rates in three-to-six-month 
programs. Similarly, completion and job placement rates and test 
score results at schools that serve highly disadvantaged students 
will be lower than those serving more advantaged students. In 
the absence of a method to adjust for student and other 
characteristics, decisions will be based on how well schools have 
recruited and not whether the training provided has been helpful. 
Thus, adjustment of outcome data is another way to measure the 
value added. 

One approach, similar to one described for measuring defaults, 
is to base assessment of a school's performance on the difference 
between (1) the actual rates for job placement, occupational 
competency, and program completion; and (2) a prediction of those 
scores based on students' economic and educational background. 
If some portion of payment to a school were contingent on meeting 
these performance standards, schools would have an incentive to 
admit only students who they believed could benefit from the 
training offered. 



Performance Funding . Performance measurement is likely to be 

most meaningful when outcomes are tied to funds. If the 

financial consequences of poor performance are indistinguishable 

from the consequences of superior performance, the system is 

likely to be perceived as irrelevant. In the context of student 

aid, performance funding could take several forms: 

c Eligibility . Minimum standards could be set for schools 
seeking eligibility for student aid. Eligibility could 
be removed if a school falls below these standards. 

o Performance contracting . Some portion of student aid 
funds could be held back, contingent upon student 
completion, placement, and retention in a job for 90 
days. This practice would be similar to JTFA 
performance contracting and would probably require 
special financial arrangements with banks or guarantee 
agencies. 

o Consumer information . A well-developed consumer 
information system could have a major effect on the 
reallocation student aid resources from institutions 
that were performing poorly to better ones. 

o Deregulation . Schools that do well on measures of 
student placement, learning and completions could be 
rewarded with fewer reporting or audit requirements. 



Phasing in the System 

Development of a fully operational performance based system 
will take several years, and will require periodic fine-tuning 
thereafter. A comprehensive system would be designed to meet 
several goals: setting minimum standards, encouraging program 
improvement, providing useful consumer information, and assessing 
the system as a whole. It would employ multiple indicators to 
evaluate a school's performance, weigh relevant background 
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characteristics of students, and rely on data that is not readily 
subject to self-serving manipulation. 



Ways to phase in such a system over a four-year period include 
the following: 

o Establish an external performance indicators panel — year 1. 

o Measure defaults adjusted for student background 
characteristics — year 1. 

o Initiate pilot projects with three states and three 

accreditation agencies to develop, test, and demonstrate 
performance measurement systems in accordance with 
student outcomes. Results should be tied to state 
licensing and accreditation. Provide planning grants in 
year 1, followed by grants to support start-up 
implementation in years 2 and 3. 

o Replace default rates with a composite default index 
based on multiple measures of default — year 2. 

o Develop statistically valid indicators to identify 

potential financial abuse and institutional insolvency. 
Tie investigations and, ultimately, base recertif ication 
on results — year 2. 

o Create a national standard for program completion on the 
basis of data that schools are required to report under 
the Student Right to Know Act; exempt schools subject to 
standards developed through pilot projects. Add other 
standards based on job placement and state 
licensure/certif ication pass rates, if such information 
becomes available through that Act — year 3. 

o Following evaluation of the pilot projects, require 
states/or accreditation agencies to adopt performance 
measures and standards that measure labor market 
outcomes, student learning, and program completions — 
year 4. 



Conclusions 

Like many other federal programs, student aid is moving toward 
a performance-based strategy to evaluate schools that provide 
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education and training. The elirination of institutions with 
high default rates from the student aid program constitutes only 
a rudimentary system of performance indicators. 



This report reviews alternative indicators of school 
performance that would be meaningful in the context of student 
aid, and the different purposes for which indicators may be 
employed. The report is limited to performance indicators that 
are appropriate for assessing proprietary school perfor-mance, 
although the approaches discussed are applicable to other types 
of institutions as well. 

Recent legislation requires that high default institutions be 
eliminated from the student loan program. The use of gross 
default rates to measure a school's performance is motivated by a 
need to reduce the large financial losses caused by defaults and 
by a belief that high-default institutions are unlikely to 
provide good quality education. The default rate indicator is 
also attractive because it can be readily implemented with 
existing data. A significant limitation is that this measure 
strongly reflects student background characteristics that alsc 
affect the propensity to default. Another problem is that 
default rates focus exclusively on the frequency of default and 
not on other important dimensions of the problem: the number of 
loans, school dependence on loans, and the amount of money in 
default. 
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Although default rates represent a short-run response to valid 
and significant problems facing the student aid program, several 
ways to enhance ED's current approach are suggested. A composite 
default index could be developed that, in addition to the student 
default rate, considers the dollar default rate and the extent to 
which an institution depends on federal student loans (percentage 
of borrowers under the Guaranteed Student Loan Program) . 

Given the very strong effect that student characteristics have 
on default, an adjustment model could be developed that holds 
schools accountable where actual loan defaults exceed a level 
predicted on the basis of student characteristics and compares 
default rates at schools serving similar populations. Much of 
the information needed to begin such an approach to defaults is 
available or can be obtained/^ 

This report reviews the role that financial risk indicators 
can play in making decisions about initial certification, 
recertification, and program review. Financial risk indicators 
are important to protect students from insolvent institutions and 
to protect taxpayers from abuse of Title IV funds. Much more can 
be done to enforce existing rules in determining eligibility for 
student aid. 



47. Information that would need new data collection includes the percentage of borrowers at a school and the extent to 
which borrowers are ATB students. 
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Because federal liabilities mount rapidly once institutions 
have entered the student aid program, more frequent and timely 
review to identify potential abuse of student aid funds is 
warranted. This requires improved data systems to provide more 
timely data (e.g., on GSL volume, defaults, and percentage of 
students on financial aid) and to provide information that is not 
currently available (e.g., the adequacy of financial arrangements 
if school fails, percentage of ATB students, percentage of school 
revenue froi. federal student aid, and percentage of revenues 
spent on instruction and recruitment) . 

It is also recommended that ED develop a statistically valid 
high risk profile to identify potentially insolvent institutions 
and institutions likely to misuse the student aid program. Much 
can be done to implement these recommendations with existing 
data, but new data requirements and statistically valid 
indicators could take two to three years to develop and put in 
place. 

This report concludes that student aid should ultimately move 
in the same direction as other vocational education and job 
training programs — toward accountability based on student 
outcomes. Three broad types of outcomes are discussed: labor 
market outcomes (e.g. training-related placements, job retention, 
wages, employer satisfaction); learning outcomes (e.g., passing 
state licensure/certif ication exams, other tests of occupational 



competency); and program completion outcomes (e.g., overall 
completion rate, aided and nonaided student completion ratio, ATB 
and non-ATB completion ratio) . 

Labor market outcomes, student learning, and program 
completions can each be measured to estimate institutional 
performance at one point in time or to indicate program 
improvement by comparing change over time. Measures of 
institutional performance at a particular time are most useful in 
performance indicator systems designed to set minimum standards 
or to provide consumer information. Long-term program 
improvement measures are more commonly employed to encourage all 
institutions to improve and to allocate rewards on the basis of 
school performance. 

Labor market outcomes, student learning, and program 
completion are all important indicators by which proprietary 
school performance may be measured. A balanced system should 
include measures in each of these areas, although not necessarily 
all the examples discussed here. 

In contrast to the data needed to measure defaults and 
financial risk, virtually none of the information required to 
measure student outcomes is currently available to ED. Although 
the Student Right to Know Act requires schools to report 
completion rates beginning in 1993 and feasibility studies are 
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underway to determine whether job placement and pass rates on 
state licensing exams can be obtained, much work will be required 
to define outcomes, to develop measures, to specify standards, to 
establish data collection systems, and to ensure the accuracy of 
information collected. 

Assessment of schools on the basis of student outcomes is a 
long-range project, although some elements could certainly be 
phased in sooner (completions, placements) than others 
(occupational competency) . A strategy for phasing in such a 
system including the initiation of pilot demonstration programs 
involving states and accrediting organizations is outlined. If 
effectively implemented, an emphasis on student outcomes could 
complement and perhaps replace default rates and financial abuse 
as primary indicators of school performance. 

The role that program inputs (facilities and equipment, 
faculty training, curriculum, instructional methods, support 
services, and student assessment) may play in a performance 
indicator system has also been reviewed. No evidence was found 
of a clear and consistent relationship between various school 
inputs, when measured quantitatively, and school effectiveness. 
In most cases the only meanirsgful way to consider school inputs 
is through on-site program reviews, such as those conducted by 
accrediting agencies and some states. Such reviews are 
inherently subjective and do not produce quantitative results 
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that permit systematic comparison across institutions. Future 
improvements in this area are unlikely to come from new 
quantitative measures, but will probably come from changes in the 
organization and procedures by which accreditation and state 
licensing are conducted. The nature of those changes is beyond 
the scope of this report. 

It is strongly recommended that ED go beyond primary reliance 
on one indicator of program performance — default rates — to a 
system that contains multiple measures. Multiple indicators 
provide a comprehensive picture of school performance, minimize 
the effects of measurement error in any single dimension, and 
discourage any unintended or perverse responses designed to "get 
around" a single indicator of performance. 

Using many indicators, however, can undermine a performance- 
based system by increasing complexity, burden, and duplication. 
This report has discussed a large number of illustrative 
indicators. An efficient system would use a limited number of 
basic indicators representing different outcomes. 

Experience demonstrates that performance indicators have the 
potential to create unfair comparisons among schools serving 
different types of students or to encourage creaming (the 
enrollment of students who are least likely to default or easiest 
to place in jobs) . Creaming could reduce the access of 
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disadvantaged students to postsecondary training-an outcome 
contrary to a basic objective of the Higher Education Act. 
Adjusting expected outcomes for the characteristics of students 
would reduce the likelihood of creaming. On the other hand, it 
could lead to lower expectations for the performance of certain 
groups of students. Setting standards that require schools to 
substantially exceed predicted levels of performance would create 
demanding expectations and provide a tool for program 
improvement. There is ample evidence from JTPA and elsewhere of 
different ways in which this may be done. 

Movement throughout the federal government toward performance- 
based systems raises some problems that are commonly faced by all 
programs that involve vocational education and job training. 
Designing ways to gain better access to state wage record data to 
determine whether job placements are related training, and to 
measure occupational competencies would be most efficiently 
undertaken through a joint inititiative by the Departments of 
Education, Labor, and Health and Human Services. 

Finally, although this paper has not recommended a specific 
government strategy for implementing performance based policies, 
in most cases, ED will be unable to operate such a system without 
the assistance of others. Appropriate roles need to be devised 
for states, for accreditation agencies, and for the federal 
government. In this regard, federal policy should recognize 
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other accountability systems that may exist as a result of state 
or other federal initiatives, and thus avoid unproductive 
conflict or inconsistency. For example, piggybacking on 
"completion rate" data collected under the Student Right to Know 
Act is preferable to instituting new data collection, with new 
definitions and requirements. Coordination of performance 
standards being developed under the Vocational Education Act with 
those that might develop through student aid would avoid gross 
inconsistencies in the way that job training programs are held 
accountable. 
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APPEm)IX A 



SUMMARY OF MEETINGS WITH 
REPRESENTATIVES OP PROPRIETARY SCHOOLS* 



In order to gain insights on how accreditation and 
certification operate in .different segments of the proprietary 
sector, Westat conducted three focus group meetings with leaders, 
owners, and or representatives of proprietary schools. These 
participants were selected based on recommendations from their 
accrediting agency and/or other knowledgeable persons in the 
field. The three meetings were held in the VJashington, D.C., 
area and involved the following groups: 

■ Cosmetology Schools, April 16, 1991 - Ritz Carlton, 
Pentagon City. 

■ Business and Secretarial Schools, April 26, 1991 - 
Holiday Inn-Capitol, Washington, D.C. 

■ Electronics Schools, May 3, 1991 - Holiday Inn-Capitol, 
Washington, D.C. 

These sessions involved lively discussion of many aspects of 
the operation, accreditation, certification, and measures of 
success of schools in these different segments of the proprietary 
sector. The dialogue provided many useful insights into the 
realities of performance assessment in different settings and in 
diverse occupational specialties. 

Each session was moderated by a member of the Westat Project 
Team, and featured active participation by other members of the 
Westat Team and by staff from the U.S. Department of Education. 
A professional writer serving on the Project Team took notes at 
each meeting and produced the following, written summaries of the 
most important points that were discussed. 



*Prepared by Westat, Inc. under Contract #LC89082001 
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FOCUS GROUP DISCUSSION 
WITH COSMETOLOGY EDUCATORS 



On April 16, 1991, U.S. Department of Education 
representatives, research consultants, and members of the 
Advisory Council on Cosmetology Relations in Education convened 
to discuss indicators of program quality, and accreditation and 
program review issues pertaining to cosmetology training. 



Indicators of Program Quality 

According to panel members, placement percentages, state 
board scores, loan default rates, the number of program hours 
completed, starting wages, and the number of individuals holding 
licenses are not sufficient, alone, as indicators of quality for 
judging cosmetology schools. These factors do, however, reflect 
the variability among schools and states, and among different 
proprietary school fields. 

For example, one panel member stated that a 100 percent 
placement rate may be the result of high demand for licensed 
employees to service a "hot" field, not of the quality of 
instruction. He added state board scores reveal little regarding 
program quality, because state boards are not testing current 
techniques and applications. Furthermore, many schools are 
preparing their students to enter the job market, not necessarily 
to pass an "outdated" state board exam. 

It was noted that in Texas state boards are up to date, and 
that default rates reflect the nature of the students and not the 
schools. Another panel member concurred. He stated many of his 
students were graduated, employed, and currently working, hut, 
for whatever reason, have defaulted on their loans. The rates 
may indicate something is wrong, but unlike a student- success 
indicator, they do not reflect the quality of instruction. 

The dkifault statistics often are misleading. A student who 
has missed several payments, but is currently making regular 
payments, is listed as in default. Moreover, lenders often are 
too hasty in moving loans to default and selling the paper to 
collectors. 
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Several panel members indicated states differed 
substantially regarding the length of training (which averages 
between 1,500-1,800 hours), starting wages, and curricula (some 
states base the exams on the curriculum, others test skill 
levels) . They noted the first and last of these factors do not 
influence starting salaries. Many also agreed that the number of 
registered cosmetologists does not equal the number of practicing 
cosmetologists. Accurate statistics do not exist— people hold on 
to their licenses to get discounts and individuals train and 
enter the market sporadically. 

Having noted poor indicators of program quality and that a 
single, national standard would be unacceptable, panel members 
agreed that key indicators should include data from student 
follow-up assessments regarding level of satisfaction and 
achievement, retention and completion rates, and employer surveys 
that measure how well the schocls are meeting their needs. 

One panel member warned against the use of a single 
statistic, or an absolute score, as an indicator of program 
quality. He would prefer a composite of different components. 
Another comment was that the industry should compile statistics 
on standard wages, number of actual working cosmetologists, entry 
level criteria, benefits, and top-earner characteristics to help 
structure indicators. One panelist added: "Rather than an 
arbitrary benchmark, send more Education Department reviewers to 
observe outcomes. Require schools to perform regular checks. 
Here is where you determine ability." 



Characteristics of a successful Program 

Successful programs meet the needs of the industi^. They 
maintain a competitive edge with the quality ol their curricula 
and ability to diversify. For example, job demand, salon 
requests, and personal referrals may indicate a field is growing 
and the need to create subspecialties to respond to a particular 
trend. Competitive schools allocate minimum hours for those who 
just want individual courses, and thereby meet students' needs. 
One panel member, who believes referrals (by former students) 
represent 70 percent of enrollment figures, said satisfied 
students indicate the school is successful. 

Although many states impose limits on the types of courses 
cosmetology schools are allowed to offer, many maintain a 
competitive edge with an enriched program. Some offer a full 
range— from basic proficiency to management training courses. 
Those states allowing discretionary hours support schools' 
efforts toward building enhanced programs. 
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According to the panel, what hurts many schools is their 
dependence on financial aid. It was argued that liberal lending 
policies are a disservice to schools and not in the best interest 
of students. 

According to another panel member, high-quality training is 
essential. Although uncertified individuals can be hired without 
training, technological improvements in the field, new chemicals, 
the need for proper instruction on the use of dangerous 
chemicals, and liability issues ensure that most employers will 
require properly trained employees. This motivates schools to 
keep their standards high. 



Accreditation and Program Review Issues 

It was related that the Commission's award of accreditation 
is good for five years. The Commission can, however, review each 
school at any time. Financial problems were not the most common 
reasons for denying accreditation. The most common offenses 
include policy, enrollment, program, and admissions issues. 

Panel members claimed significant differences exist between 
accredited and nonaccredited schools. As a result of 
accreditation policies, the industry has become sophisticated and 
schools' operations have become more structured. Also, 
established standards enable fair comparisons among ratings of 
schools. Panel members contrasted the advantages accreditation 
has brought to the industry with the shortcomings of the 
Education Department's reviews. One ED reviewer was described as 
"ensconced in the office reviewing financial aid files." With 
such a limited focus on the school's ability to follow 
requirements (such as posting, signing, and handling financial 
aid refund forms) , reviewers cannot appreciate and monitor the 
broader and more meaningful characteristics defining each school. 

One panel member complained of too many audits from too many 
agencies. Another recommended cooperation between governing 
units. He believes quality audits are best addressed by the 
accrediting board and financial audits by the Department of 
Education. "We need mutual reinforcement that recognizes 
separate functions." 
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Recommendations for Lowering Defaiilt Kates 

1. Develop a composite profile of high-rick students using 
factors such as family background, level of education, 
income level, and number of dependents; and compensate 
schools for servicing these high-risk students. 

2. Allow lending agencies and the government to support the 
school's efforts to collect from defaulters, by 
disclosing pertinent information. 

3. Develop more sophisticated default measures; recognize 
the size of the loans with the default rate. 

4. Alter the program, which currently requires repayment to 
begin six months after graduation, which for many is too 
short an interval. 

5. Encourage loan-servicing agencies to maintain the loans 
or at least keep them in-state. 

FOCUS GROUP DISCUSSION 
WITH REPRESENTATIVES OP BUSINESS AND SECRETARIAL SCHOOLS 

On April 26, 1991, U.S. Department of Education 
representatives, research consultants, and representatives of 
business and secretarial schools convened to discuss indicators 
of program quality, and the roles of the federal government, 
accrediting agencies, and state licensing authorities in 
assessing the quality of institutions. 

Indicators of Program Quality 

According to the three panel members, placement and default 
rates of proprietary institutions are poor indicators of program 
quality at business and secretarial schools. Members said these 
rates do not take into account the effects of environmental and 
cultural factors; and the term ''proprietary" is not defined with 
sufficient precision to accurately reflect the status of business 
and secretarial schools. 

For purposes of this discussion, a proprietary institution 
was defined as a "profit-making institution that offers short- 
term, occupationally oriented, vocational and technical 
training." Nevertheless, panel members insisted that programs 
should be divided further according to factors such as public or 
private control; degree or nondegree granting; region; type of 
training; educational mission (occupational or educational goal) ; 
and length of program. 
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One panel member also compared characteristics of court 
reporters, diesel mechanics, truck drivers, and nursing 
assistants to show how cultural factors and job occupations, not 
program quality, affect placement and default rates. He said 
enrollees in court reporter programs typically aspire to a 
middle-class status and have a high drop-out rate due to the 
English language proficiency and manual dexterity requirements of 
court reporting. (According to several panelists, the 
institution cannot screen for those who are destined to "top out" 
on a needed skill half-way through the course.) These students 
also have low completion, high placement, and low default rates. 
On the other hand, diesel mechanics also are well placed, earn a 
good salary, and repay their loans, he said. Truck drivers come 
from farm/rural areas, have a good placement rate, but a high 
default rate. And, nursing assistant students, who typically are 
single-parents, with an inner-city and welfare heritage, have a 
high dropout rate. One discussant also cited a study that showed 
default rates do not differ between those who drop out from the 
nursing assistant program and those who are placed. 

The panel agreed that indicators of program quality are 
needed; however, they believe the Department of Education (ED) 
should first categorize institutions fairly and then only compare 
default rates of schools serving similar students. ED also 
should require annual, independent, certified public accountant 
audits, said one participant. Panel members agreed that fiscal 
responsibility and faculty competence are important factors in 
evaluating the quality of an institution. 



Roles in Assessing Quality 

One panelist asserted it is the federal government's 
responsibility to set standards for student outcomes, learning, 
and minimum thresholds. Because ED provides the money, said 
another panelist, it should set the standards and regulate 
compliance. Another was opposed to instituting national tests to 
set a standard for quality because of the variability within and 
among occupations. He noted that the tests used within schools 
to monitor progress and certify completion do not necessarily 
measure the skill that will be used on the job. One panelist 
warned participants that imposing occupational objectives on 
institutions that are degree-granting will not work, but added, 
this could be clarified in ensuing legislation. 

Panel members said data for setting placement ranges and 
enabling valid comparisons between similar institutions exist and 
are easily accessible to the government's agencies. The federal 
government need not devise new federal standards. 
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According to one panelist, the roles of state licensing 
authorities and accrediting agencies are not mutually exclusive 
in spite of their differences. He characterized licensing as a 
static process that is administered externally and measures a 
minimal level of compliance. Whereas the licensing agency should 
be responsible for constant monitoring, said one panelist, 
accrediting agencies should take periodic "readings" to assess an 
institution's status. He said the accreditation process is 
dynamic and operates on peer review, not a system of minimums. 

Another panelist said the accreditation process forces 
institutions to develop long-range, strategic plans. These plans 
can be developed into workable documents that provide criteria 
(qualitative, albeit subjective, data) for determining the 
quality of an institution. But another participant believes 
industry, not accreditation agencies, is better able to set the 
indicators of quality. He described three types of accreditation 
(program, regional, institutional) and two administrative 
operations — a certification agency to read and ensure proper 
execution of administrative and financial regulations and an 
eligibility agency, which addresses an institution's form and 
structure . 

Continuing Concerns 

Panel members reiterated their frustration at ED's use of 
the term "proprietary" to describe business and secretarial 
schools along with institutions that ere fundamentally, 
characteristically, and operationally different. They also 
reexpressed their concern that the federal government does not 
take into account mitigating factors such as the nature of the 
student population and the location of schools when 
characterizing institutions with default rate data. 

Recommendations for lowering the rate were not forthcoming; 
however, one panelist offered his belief that servicing agencies 
and lenders are part of the problem of high default rates. For 
the following reasons business and secretarial schools should not 
be penalized for their default rates: 

1. Schools have no control over who gets a loan. They are 
unable to screen high-risk applicants. 

2. Schools are precluded from learning about the students in 
default. 

3. Students who were in default and began repayment are 
still considered in default. 

4. Schools do not decide when the money is awarded (GSL 
borrowers, who have a lower dropout rate receive their 
money 30 days after school begins.) 
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One of the panelists provided an extensive set of position 
papers and data on default rates for different types of business 
and secretarial schools, definitional issues relating to the term 
"proprietary," and other related matters. 



FOCUS GROUP WITH 
DIRECTORS OP ELECTRONICS SCHOOLS 



On May 3, 1991, a U.S. Department of Education 
representative, research consultants, and directors of 
electronics schools convened to discuss indicators of program 
quality and factors that help and hinder school operations. 

Progrzun and Student Profile 

According to the directors of electronics schools, these 
institutions offer a variety of programs that can be completed 
within six to 24 months. Upon completion a graduate can earn a 
certificate or Associate of Arts degree. Not all schools require 
admissions testing; however students must have a high school 
diploma or GEO to enroll. Some schools require their students to 
take a final, comprehensive examination at the end of the 
program. Other schools prepare the students and encourage them 
to take an exit test for occupational competency. 

The differences between schools with six-month programs and 
those with 2 4 -month programs are not reflected in the placement 
rates of graduates, said one director. According to another 
director, neither is there a difference in the marketability of 
students completing the 12- and 18-month programs. He added, the 
Associate of Arts degree is designed to help graduates earn 
promotions, after they have been hired. 

The directors described electronics students as needing 
extensive support in nonacademic areas. Often the students were 
unable to survive or learn in traditional secondary educational 
settings. The electronics students considered most likely to be 
graduated and placed are white, male, single, recent high school 
graduates. Panel members said older, female, and minority 
students are less likely to complete their training because of 
familial and financial concerns. 

One director said employers want to hire minority and female 
graduates. However, another noted that those with families often 
are more ifficult to place, because many jobs require 
relocation. One directors' recommendation for increasing 
retention and graduation rates is "give these people grants, and 
child care, and work study— no loans." He added the loan system 
has numerous problems that create additional barriers for the 
already overburdened student. 
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Characteristics of "Good" Schools 

Two panelists maintained that there are no "good" or "bad" 
schools; the terms describe people. They stressed that 
proprietary schools in general have a poor image and that few 
observers recognize the tremendous work involved in training and 
ensuring gainful employment for a segment of the population that 
is labelled "the refuse of the secondary system." This image is 
exacerbated when success stories and anecdotal information are 
ignored and only the data based on maintaining accountability in 
traditional school systems are reported. 

Another director concurred, however he noted that good 
schools have an admissions office that is geared to be an 
advocate for prospective students. He said admissions testing is 
a tool that helps assess the probability of the school's success 
and the potential of the student. According to this panelist, 
employers test prospective employees as well as set minimum 
competencies. Therefore, successful schools must be market 
responsive. This director criticized schools that focus on 
advertising and recruitment, but lack libraries and sound safety 
programs . 



Indicators of Program Quality 

Panel members said default, completion, placement, and 
starting wage rates do not indicate quality of electronics 
programs because they are not "sufficiently broken down." These 
rates blanket critical data describing their schools and student 
populations and do not account for idiosyncrasies (such as 
program choice) in the field. For example, one panelist said 
automotive students tend to have lew family income, high 
financial aid needs, high default rates, and high drop out rates. 
One may not expect these individuals to succeed, yet they still 
find employment. 

The directors said factors such as family income and 
retention on the job affect the default rate, but they really 
predict student behavior and do not indicate school quality. To 
assess quality, one panelist recommended that reviewers focus on 
growth potential instead of starting wage. A graduate might earn 
a low wage initially , but can do very well within two years, said 
another director, because many employers have their own training 
programs. They specifically look for trainability, appearance, 
and a foundation in electronics. They then train their employees 
to operate their equipment. One director also added, that where 
possible, articulation ability should be assessed. In areas 
where colleges are proximal, it would indicate that the 
curriculum is worthy of transfer. 
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One of the panelist believes evaluations of a school's 
admissions process; the educational background of the faculty; 
management, staff, and faculty turnover rates; and a school's 
ability to keep pace with changes in technology and machinery can 
serve as indicators of program quality. Another advocated a 
"value added" approach, which involvns assessing the student upon 
enrollment as well as upon graduation. He suggested other 
indicators such as faculty-student ratio and the viability and 
longevity of schools. 



Agency Roles 

According to one director, accreditation bodies are 
interested in quality, not regulations. And although another 
director thinks the accreditation system works, he also thinks 
electronics schools are overregulated because of state, federal, 
and sometimes county guidelines that must be observed. One 
panelist recommended that the governing/reviewing agencies 
synchronize their reporting requirements and share appropriate 
information without being competitive. Two directors remarked 
that state licensing agencies lack the authority to "do anything 
about schools they do not want to license." 



KecGmmendations for Lowering Default Rates 

The default rate is not a school problem, it is a student 
problem, said one director. Published default rates, which 
always are a few years behind, do not break down to the 
contributing elements, and worse, they mask other problems. For 
example, one director said 40 to 50 percent of his students come 
from other secondary institutions. If any of these students 
defaults on a loan, even a loan incurred at the former school, 
that student is listed as a default statistic of the electronics 
school (i.e., the students' deferment becomes the electronics 
schools' default statistic). 

To simply lower the default rates, it was suggested that 
schools can (1) change admission standards; (2) run credit tests; 
and (3) change their programs. However, these measures would bar 
the high-risk student, the one most in need of financial aid and 
social support, from an opportunity to become a productive member 
of society. 
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The directors suggested a variety of ways schools, financial 
agencies, and the Department of Education can help lower default 
rates . 

The panel recommended that schools: 

1 . Implement a default management program (i.e., send 
registered letters (reminders) after the student is 
graduated, and followup 30- and 60-day reminders) . 

The panel recommended that lending agencies: 

1. Set disbursement schedules to correspond with the periods 
of attendance; and 

2. Make the students pay. 

The panel recommended that ED: 

1. Tighten the branching procedures and policies; 

2. Ensure that loan and service agencies are close to the 
school (if possible); and 

3. Scrutinize cooperating financial institutions (are all 
loans coming from or being serviced by one institution?) . 
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OVERVIEW OP OCCUPATIONAL COMPETENCY TESTING 
IN PROPRIETARY SCHOOLS^ 



Proprietary schools typically do not administer 
standardized occupational proficiency tests to their 
students. Testing is, however, carried out to state 
licensing agencies in many occupations for which proprietary 
schools provide training. In addition, increasingly states 
are developing occupational cc>mpetency tests for trades 
taught in state vocational technical education programs. 
Many of the same occupations are also taught in proprietary 
schools. This report discusses the availability of tests 
from both these sources. The report also discusses research 
on the reliability of occupational competency tests and 
several caveats offered by researchers on the use of these 
items in developing standards of performance. 



Testing Through State Licensing Agencies 

The states oversee more than 1,000 occupations and 
professions through licensure, certification, the number of 
occupations licensed varies from state to state, or simple 
registration. The range of these occupations is 
extraordinarily diverse, and in the field of cosmetology 
alone, more than 20 different specialties are regulated in 
one state or another. For example, in cosmetology, every 
state requires a license for cosmetologist/hairdresser but 
only one - Oregon - licenses facial technicians. 

The picture is similarly diverse with respect to the 
prevalence of competency testing as part of the licensing 
process. Again, different states test in different 
occupations. No national data are available on the number 
of occupations in which licensing tests are carried out, so 
it is difficult to determine how many states give licensing 
tests for occupations taught in proprietary schools. As 
discussed later, the available data do allow some broad 
generalizations to be drawn about the prevalence of 



1. Prepared by Westat, Inc., under Contract #LC89082001. 



2. Brinegar, Introduction and pp. 26-28. 
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competency testing by broad program area. 

The distribution of 1988-89 proprietary school enrollment, 
by program area follows.^ 

Percentage of 
Proprietary 

Program Area Students Enrolled 



Word processing, secretarial, and 
other business/marketing 

Cosmetology, barber ing, and 14 
other personal services 

Medical, nurse assistant, and other 
health care occupations 

Computer, electronics, and other 
technology 

Construction, welding, and other 
trades/ industrial 

Truck driving and other transportation 
Other 



100 



According to a 1990 survey by the National 
Clearinghouse on Licensure, Enforcement, and Regulation, no 
states regulated word processing, secretarial, or related 
occupations, and only a few states regulated technology 
occupations/ Thus, for the two largest proprietary school 
fields, which represent more than two-thirds of enrollments, 
state licensing examinations are largely unavailable. 



3. Apling, p. 13. 



4. Brinegar, Introduction and pp. 26-28. 
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14 

7 
25 
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Less than 1 
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In those proprietary school occupations in which the 
states do test, there is little national standardization. 
Test philosophies and composition vary from state to state, 
as illustrated by two of the states reviewed in this study: 
Pennsylvania and Colorado. Since 1980, Pennsylvania has 
required that all licensing tests be developed, prepared, 
scored, and administered by qualified third party test 
developers. In the late 1970s after becoming concerned 
about the validity and reliability of state board exams, 
Pennsylvania decided to take test development and 
administration out of the hands of the state boards and to 
contract the exams out to private test development firms. 
The state believes that performance tests are less valid 
than written, multiple-choice tests for measuring 
occupational competency and thus only written tests are used 
in Pennsylvania. Colorado, by contrast, maintains a 
centralized test development agency in the state government, 
and nearly all the state's licensing tests are developed by 
that state agency. Also, unlike Pennsylvania, Colorado uses 
performance tests in conjunction with written tests. 

There are a few exceptions to this prevailing lack of 
standardization. In several proprietary school occupations, 
national associations have developed competency tests used 
by many states. These occupations include: cosmetology 
(Interstate Council of State Boards of Cosmetology); nurse's 
assistant, aide (National Council of State Boards of 
Nursing) ; welding (American Welders Society) ; construction 
occupations (American General Contractors ' Association) ; and 
automotive repair (Automotive Service Institute) . 



Testing Under State Vocational/Technical Education 
Programs 

An increasingly important source of standardized 
occupational competency tests is the state vocational 
technical education programs. According to a 1989 report of 
the Office of Technology Assessment (OTA) , the states have 
become very active in the development and use of 
occupational competency tests, largely in response to the 
increasing emphasis on performance measurement and 
accountability in education and training programs. 
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According to a 1988 survey, 13 states were testing the 
occupational competencies of vocational and technical 
students and an additional 18 states were either in the 
process of developing tests or considering development (see 
Figure 1) . In contrast to the examinations given by state 
licensing agencies, vocational technical education 
achievement tests include achievement tests for occupations 
commonly taught in proprietary school such as secretarial, 
office, and technician occupations. Analysis of the 
examinations would be required to determine whether they 
would, in fact, be appropriate for measuring achievement in 
proprietary schools. 

Many states have formed consortia to pursue the cost- 
effective development of competency-based curricula and 
tests. Two of these consortia are the National Occupational 
Competency Testing Institute (NOCTI) and the Vocational 
Technical Consortium of States (V-TECS) . According to its 
recent materials, NOCTI has developed written and 
performance competency tests in more than 60 secondary and 
postsecondary fields; V-TECS has developed tests in 24 
occupations and has 18 more under development. In addition 
to consortia-developed tests, several states have taken the 
initiative in developing their own tests. A notable example 
is Oklahoma, which has one of the most advanced competency- 
based curriculum and testing systems in the country. 
Oklahoma has developed test batteries of 168 occupations, 
each of which has three components: a traditional multiple- 
choice test, a test of decision-making skills within the 
occupation, and a performance test. 



Reliability of Occupational Competency Tests 

In its review of the state of the art in occupational 
competency testing, OTA found evidence that specific 
occupational competency tests correlate strongly with job 
performance. One study cited in the OTA report synthesized 
the results of 262 studies of occupational competency tests 
and concluded that there was a high correlation between test 
scores and supervisors' ratings of job performance. The 
correlation was higher than that of other predictor 
variables such as ability tests, interviews, and 
biographical inventories. According to OTA, other studies 
have reached similar conclusions. 
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However, OTA cautions against overreliance on 
occupational competency tests and argues that they should 
not become the sole basis for performance standards. Among 
the reasons for caution were these: 

■ It is never possible to know how much of an 
individual's abilities can be attributed to a school 
or program. Socioeconomic and other correlates of 
test performance may also be factors. 

■ There is a risk that schools will coach students on 
test-taking strategies at the expense of actually 
teaching the occupational skills themselves. 

■ Overt ».liance or occupational competency tests could 
resulv in overemphasis on learning highly 
specialized skills at the expense of more broadly 
applicable generic skills. 

Furthermore, one of the advantages that proprietary 
schools claim over public vocational technical training is 
their ability to adjust quickly to marketplace demands. As 
employers request training in new technologies and methods, 
proprietary schools adjust the content of their programs. 
It is doubtful that occupational competency tests can adjust 
so quickly to test these new skills. 
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